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Abstract 

Contract  cost  growth  has  been  a  concern  for  the  Department  of  Defense  (DoD)  for 
decades.  Earned  value  management  is  a  tool  used  by  the  DoD  to  assist  in  identifying  cost 
overruns  before  they  occur.  Current  DoD  regulations  require  contracts  to  report  their 
earned  value  management  (EVM)  data  down  to  level  three  of  the  work  breakdown 
structure  (WBS).  Previous  research  has  shown  level  three  EVM  data  can  predict  contract 
cost  growth  earlier  than  using  level  one  EVM  data.  This  research  examines  if  level  five 
EVM  data  would  better  predict  cost  growth  than  level  three.  The  results  indicate  that 
level  five  is  not  a  better  predictor  of  cost  growth  then  level  three.  Thus,  the  results  do  not 
support  the  DoD  requiring  contractors  to  provide  level  five  EVM  data. 
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COMPARING  THE  PREDICTIVE  CAPABILITIES  OF  LEVEL  THREE  EVM 
COST  DATA  WITH  LEVEL  FIVE  EVM  COST  DATA 


Chapter  1  -  Introduction 


Background 

The  Department  of  Defense  (DoD)  outsources  most  major  acquisitions  projects  to 
different  private  companies;  so  it  is  imperative  for  the  DoD  to  have  effective  oversight  of 
these  projects.  Earned  value  is  the  method  the  DoD  has  decided  to  use  in  order  to 
achieve  this  oversight.  In  1967,  the  DoD  created  the  Cost/Schedule  Control  Systems 
Criteria  (C/SCSC).  The  C/SCSC  used  thirty-five  different  formulas  to  assist  program 
managers  and  commanders  in  determining  if  a  project  was  on  budget  and  on  schedule. 
The  Earned  Value  Management  System  (EVMS)  replaced  C/CCSC  in  1997. 

Earned  value  management  has  been  successful  in  assisting  program  managers  and 
commanders  with  oversight  of  programs  for  over  fifteen  years.  As  in  any  profession,  the 
evaluation  and  rewards  of  contracts  depend  upon  the  contractor’s  performance. 

However,  there  are  some  concerns  with  EVMS  as  well.  Some  contractors  learn  how  to 
beat,  or  at  least  go  around,  the  system.  Some  project  managers  learn  how  to  manipulate 
their  earned  value  numbers  in  order  to  have  their  programs  appear  on  time  and  budget, 
when  in  reality,  the  program  manager  is  aware  the  program  is  not  (CIO  Insight,  2005). 
For  example,  if  a  project  manager  says  a  task  is  80%  complete,  EVMS  might  allege  the 
project  is  on  time  and  on  budget.  However,  what  proof  is  there  that  the  task  is  80% 
complete,  not  just  40%  complete?  Slight  changes  in  the  completion  percentage  can  make 
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a  large  impact  on  the  actual  status  of  a  project  (CIO  Insight,  2005).  This  issue  of 
intentional  misrepresentation  will  always  be  a  concern  for  the  DoD. 

Another  concern  for  the  DoD  with  EYMS,  not  involving  intentional 
misrepresentation,  is  what  level  to  report  cost  data  using  EYMS  formulas.  There  are 
currently  three  techniques  to  evaluate  cost  data:  a  top  level  showing  the  total  project  cost, 
a  bottom  level  breaking  the  cost  down  to  the  lowest  level  (control  account),  or  the  third 
option  of  somewhere  in  between.  Each  of  these  three  techniques  has  positive  and 
negative  aspects,  respectively. 

Using  top-level  cost  data  (Work  Breakdown  Structure  (WBS)  level  1),  the  project 
manager  is  able  to  obtain  a  quick  picture  of  the  overall  status  of  a  project.  The  negative 
aspects  of  this  approach  are  that  the  overall  project  might  currently  be  on  budget  and 
schedule,  even  if  there  is  a  lower  level  element  that  is  falling  behind  schedule  or  over 
budget.  This  could  cause  the  entire  project  to  fall  behind  or  go  over  budget.  Because  the 
purpose  of  EVMS  is  to  predict  future  problems,  top  level  cost  data  would  show  only  the 
current  state  of  the  project  with  limited  predicative  capabilities. 

Using  bottom  level  cost  data  (control  accounts)  is  a  practice  widely  used  by 
companies  in  the  private  sector  who  use  EVMS  (DAU  Website,  2013).  Using  this  level 
of  cost  data  allows  the  project  manager  to  see  exactly  what  area(s)  have  the  greatest  risk 
of  going  over  budget  or  falling  behind  schedule.  The  disadvantages  of  using  these  cost 
data  is  that  the  project  manager  now  has  to  track  many  different  data  points,  and  it  drives 
up  the  management  cost  due  to  the  extra  accounting  work  and  man-hours  to  prepare  the 
data. 
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Currently,  the  DoD  uses  the  third  method.  The  current  requirements  are  for 
contractors  to  report  cost  down  to  work  breakdown  structure  level  three.  However,  there 
are  some  exceptions  to  this  rule.  High-risk  or  high-cost  elements  in  a  project  require  a 
lower  collection  of  cost  data.  The  reason  for  requiring  this  level  is  that  the  government 
“does  not  want  to  constrain  the  contractor’s  ability  to  define  or  manage  the  program  and 
resources”  (MIL-STD-881C,  2011). 

Purpose  of  Research 

The  purpose  of  this  research  is  to  determine  the  predictive  nature  of  EYM  when 
the  government  has  visibility  of  cost  data  down  to  level  five  of  the  WBS.  Currently,  the 
government  requires  its  contractors  to  provide  cost  data  only  at  level  three  of  the  WBS. 
Knowing  there  are  additional  costs  associated  with  obtaining  additional  information  from 
the  contractor,  is  it  beneficial  for  the  government  to  pay  the  additional  cost? 

The  cost  of  any  DoD  contract  can  be  broken  down  into  two  main  areas,  the  Total 
Allocated  Budget  (TAB)  and  profits  and  fees.  The  TAB  is  the  sum  of  all  budgeted  costs 
for  a  contract.  The  dollar  value  of  the  TAB  can  fluctuate  throughout  the  contract  life  and 
on  contracts.  The  Budget  at  Complete  (BAC)  and  the  TAB  can  be  interchangeable  on 
contracts  (DAU  Gold  Card,  2013).  Figure  1  displays  the  relationship  between  the 
contract  price  and  its  components.  Found  in  Appendix  A,  are  a  complete  list  of  EYM 
acronyms. 
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Figure  1-  Contract  Price  Diagram 

The  work  breakdown  structure  (WBS)  divides  all  government  projects  (MIL- 
STD-881C,  201 1).  The  WBS  divides  the  overall  project  into  multiple  layers  of  smaller 
sub  components,  which  allows  the  government  to  monitor  and  track  what  tasks,  or  parts, 
are  required  to  complete  the  project.  However,  contractors  also  use  an  organizational 
breakdown  structure  (OBS).  The  OBS,  like  the  WBS,  divides  the  different  parts  of  a 
project,  but  instead  of  breaking  the  different  parts  down  by  product,  the  OBS  breaks  the 
parts  down  by  which  department  of  the  organization  will  actually  work  on  the  part.  The 
control  account  is  where  the  WBS  and  OBS  meet.  Figure  2  illustrates  a  generic 
relationship  between  the  WBS  and  the  OBS.  However,  the  figure  gives  the  impression 
that  the  control  account  is  located  at  level  five  of  the  WBS,  which  might  not  be  true. 
Every  program  will  have  different  levels  in  its  WBS.  The  DoD  has  attempted  to 
standardize  only  the  first  three  levels  of  the  program  WBS. 
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Figure  2  -  CWBS  and  OBS  Intersection 

This  research  looks  into  the  comparison  of  earned  value  management  at  level  five 
compared  to  the  current  reporting  standard  of  level  three  of  the  WBS.  Currently,  when  a 
contractor  reports  the  status  of  a  project  to  the  government,  the  contractor  is  required  to 
show  the  project’s  cost  information  down  to  the  third  level  of  the  contract  WBS. 

Program  managers  and  analysts  use  cost  information  to  predict  if  the  project  is  on 
schedule  and  budget  in  various  EVM  formulas.  This  research  investigates  if  it  would  be 
more  beneficial  for  the  government  if  the  contractors  would  provide  cost  information 
down  to  level  five. 


Research  Objective 


The  main  objective  of  this  research  is  to  determine  if  level  five  EVM  data  will 


predict  overall  cost  growth  earlier  and  more  accurately  than  level  three  EVM  data.  Then 


if  level  five  is  more  predictive,  how  much  extra  would  it  cost  the  DoD  to  obtain  this  data 
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and  is  the  extra  cost  worth  any  additional  value  level  five  data  has?  Previous  research  has 
proven  the  benefits  of  using  level  three  EYM  data  compared  to  level  one  EYM  data 
(Rosado,  201 1).  However,  due  to  inconsistencies  in  the  data  available,  Rosado  was 
unable  to  research  below  level  three. 

Research  Questions  #1 

The  primary  question  is  if  level  five  EVM  data  is  more  predictive  than  level  three 
EVM  data  at  predicting  overall  contract  cost  growth.  If  level  five  data  is  not  more 
predictive,  the  government  should  not  invest  additional  money  into  the  contract.  To 
answer  this  research  question  we  evaluated  the  following  three  investigation  questions: 

Investigation  Question  #1.1 

Which  EVM  metric  should  we  use  to  predict  cost  growth?  Rosado  used  the  EAC 
provided  by  the  contractor  to  compare  cost  growth  while  Keaton  also  included  the  cost 
performance  index  (CPI)  and  schedule  performance  index  (SPI). 

Investigation  Question  #1.2 

Our  last  investigating  question  for  research  question  #1  is  can  we  build  a 
statistical  model  using  levels  three  and  five  EVM  data  to  predict  overall  contract 
performance? 

Research  Questions  #2 

The  next  research  question  is  what  is  the  additional  cost  associated  with  requiring 
contractors  to  provide  level  five  EVM  data?  In  addition,  should  we  have  the  contractor 
include  all  level  five  elements  or  just  a  certain  percentage  based  on  the  size  of  each  level 
five  element  compared  to  the  overall  size  of  the  contract? 
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Methodology 

We  plan  to  use  a  regression-based  approach  to  determine  the  relationship  between 
our  level  three  and  five  EYM  data  and  the  overall  contract  cost  growth.  Regression- 
based  models  have  been  widely  used  in  previous  research  to  predict  cost  growth  (Rosado, 
2011;  Trahan,  2009;  Thickstun,  2010).  However,  Trahan  and  Thickstun  both  looked  at 
top-level  EYM  data  while  Rosado  examined  level  three  EVM  data.  We  believe  we  will 
be  able  to  look  down  to  level  five  due  to  an  improvement  in  EVM  reporting. 

The  Office  of  Performance  Assessments  and  Root  Cause  Analyses  (PARCA)  is 
the  organization  that  oversees  earned  value  management  within  the  Department  of 
Defense  (DoD)  (PARCA  Website).  PARCA  is  continually  attempting  to  improve  the 
implementation  of  EVM  data. 

PARCA's  goal  is  to  increase  earned  value's  constructive  attributes  for  the  DoD 
firms  managing  acquisition  programs  by  reducing  the  economic  burden  of 
inefficient  implementation  of  EVM.  PARCA  is  dedicated  to  the  concept  that 
EVM  is  a  management  tool,  not  merely  a  contractually  required  report  (PARCA 
Website) 

Using  the  Earned  Value  Management  -  Central  Repository  (EVM-CR),  we  have 
access  to  approximately  “30,000  Contractor  Cost  Data  Reports  (CCDR),  Software 
Resources  Data  Reports  (SRDR)  and  associated  documents  (DC ARC  Website,  2013).” 

In  addition,  from  talking  with  an  analyst  at  PARCA,  he  estimated  there  are  approximately 
177  contracts  within  the  EVM-CR  that  contain  level  five  EVM  data  (Pflieger,  2013). 

After  we  run  our  regression  models,  we  need  to  clarify  we  are  not  attempting  to 
show  causality  between  overall  contract  growth  and  our  independent  variables.  We  are 
attempting  to  show  that  the  relationship  between  our  independent  variables  and  EAC 
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growth  act  as  predictors.  Our  goal  is  to  create  a  regression  model  that  will  provide  an 
early  warning  to  project  managers  and  analysts  of  possible  cost  growth  within  a  contract. 
We  ultimately  want  to  identify  if  level  five  EYM  data  can  predict  that  growth  earlier  and 
more  accurately  than  level  three  data. 

Assumptions 

For  our  research,  we  made  two  key  assumptions  about  the  EYM  data  available. 
The  first  assumption  we  have  to  make  is  the  contracts  with  level  five  EVM  data  are  not 
biased.  As  we  stated  earlier  in  this  chapter,  contractors  are  required  to  report  below  level 
three  for  contracts  that  are  considered  high  risk  or  high  cost.  The  risk  we  are  assuming  is 
that  not  all  of  the  available  contracts  with  level  five  EVM  data  are  just  either  high  risk  or 
high  cost.  Another  assumption  we  made  was  that  the  WBS  data  available  was  contract 
WBS  not  program  WBS.  What  this  means  to  our  research  is  that  we  will  not  attempt  to 
prove  a  program’s  overall  cost  growth.  Instead,  we  will  be  looking  at  individual 
contract’s  cost  growth. 

Overview  of  Thesis  Chapters 

In  the  following  chapter,  we  will  fully  explain  the  origins  of  earned  value  and 
discuss  previous  research  completed  on  relevant  topics.  In  Chapter  Three,  we  will 
explain  how  we  obtained  our  data  and  how  we  selected  the  data  points  we  will  use  in  our 
model.  Then,  we  will  continue  to  explain  our  methodology  on  how  we  will  analyze  and 
evaluate  our  data.  In  Chapter  Four,  we  will  show  the  results  of  our  statistical  model. 
During  the  final  chapter,  we  will  give  a  conclusion  and  some  possible  follow-on  research 
problems. 
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Chapter  2  -  Literature  Review 


Purpose 

The  purpose  of  this  chapter  is  to  review  relevant  literature  dealing  with  earned 
value  management  system  (EVMS).  In  this  chapter,  we  will  define  the  major  parts  as 
well  as  look  at  the  history  of  earned  value  and  why  it  is  important.  We  will  then  discuss 
research  dealing  with  problems  of  the  effectiveness  of  the  DoD’s  earned  value  system 
operations.  In  addition,  we  will  provide  some  information  on  completed  relevant 
research. 

Define  Work  Breakdown  Structure 

Understanding  the  parts  of  a  problem  is  the  best  way  to  understanding  the 
problem.  The  first  area  we  are  going  to  explain  is  a  work  breakdown  structure  and  its 
intended  uses.  Segmentation  occurs  with  the  WBS  into  two  types  of  WBSs.  The  first 
type  is  a  Program  Work  Breakdown  Structure  (PWBS).  The  other  type  of  WBS  is  the 
Contract  Work  Breakdown  structure  (CWBS).  Both  types  of  WBS  break  down  the 
elements  of  a  project  into  logical  levels  (MIL-STD  881,  201 1). 

The  Program  WBS  shows  the  entire  program  at  a  summary  level.  The  program 
WBS  will  consist  of  at  least  three  levels  (Albert,  2008).  The  DoD  standardizes  these 
three  levels.  Contractors  use  PWBSs  to  report  the  status  of  their  projects  to  the  DoD. 
Only  the  top  three  levels  of  the  WBS  are  required  because  requiring  additional  lower 
levels  could  affect  a  contractor’s  methods  and  cause  delays  in  processes  due  to  the 
additional  reporting  requirements  (Albert,  2008).  Level  one  of  the  WBS  ascertains  the 
entire  item  usually  identified  as  a  major  program  or  a  project  within  an  aggregated 
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program.  Levels  two  and  three  recognize  the  elements  of  the  parent  level  broken  down  to 
their  next  logical  steps.  Figure  3,  from  MIL  STD  88 1C,  displays  a  simple  relationship 
between  the  first  five  levels  of  the  WBS  (MIL  STD  88 1C,  2011). 


Figure  3  -  Simple  PWBS 

The  contractor  uses  the  other  type  of  WBS,  the  CWBS.  The  CWBS  will  normally 
start  where  the  PWBS  stops  and  continues  down  to  at  least  one  level  below  the  negotiated 
DoD  reporting  level.  In  most  cases,  the  contractors  will  have  their  CWBS  going  all  the 
way  down  to  the  control  account  level.  The  benefits  of  the  CWBS  for  the  contractor  are 
that  the  CWBS  allows  the  contractor  to  break  the  work  down  into  manageable  size 
elements.  By  breaking  the  work  down  to  a  manageable  level,  the  contactor  can  track  the 
individual  people  working  on  a  project  and  determine  exact  cost  and  schedule  data  for 
each  element.  The  lowest  level  of  the  WBS  is  the  control  account.  The  control  account 
is  where  the  WBS  intercepts  the  organizational  breakdown  structure  (OBS). 
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The  main  difference  between  the  PWBS  and  the  CWBS  is  standardization.  The 


DoD  has  developed  twelve  different  standardized  PWBS.  MIL-STD  88c  contains  a 
breakdown  of  each  WBS  structure.  The  reason  for  twelve  different  WBS  is  the 
differences  between  types  of  project.  These  standardizations  assist  the  DoD  when 
evaluating  similar  projects,  indicating  one  will  have  the  ability  to  find  the  same  elements 
in  levels  one  through  three  for  a  C-130  or  an  F-35.  In  contrast,  in  a  contract  WBS,  the 
elements  can  vary  by  how  the  contractor  decides  to  break  down  each  element.  The 
standardization  reason  for  the  PWBS,  while  the  CWBS  is  not,  is  because  the  DoD  is 
responsible  for  developing  and  maintaining  the  PWBS,  while  the  individual  contractor  is 
responsible  for  the  CWBS  (Albert,  2008). 

Earned  Value 

In  this  section,  we  will  explain  what  earned  value  is  and  explain  some  earned 
value  terms  and  definitions.  We  will  also  go  into  the  need  for  earned  value  and  its 
purpose  within  the  DoD. 

Terms  and  Definitions 

Analysts  often  uses  the  following  terms  interchangeably;  earned  value  analysis, 
earned  value  management,  and  earned  value  management  system.  However,  there  is  a 
distinctive  difference  between  each  of  these  terms  (Lukas,  2008).  The  following 
definitions  are  from  Lukas’s  article  “Earned  Value  Analysis-Why  it  Doesn’t  Work”: 

•  Earned  Value  Analysis  (EVA)  -  a  quantitative  project  management  technique  for 

evaluating  project  performance  and  predicting  final  project  results,  based  on 
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comparing  the  progress  and  budget  of  work  packages  to  planned  work  and  actual 
costs. 

•  Earned  Value  Management  (EVM)  -  a  project  management  methodology  for 
controlling  a  project,  which  relies  on  measuring  the  performance  of  work  using  a 
Work  Breakdown  Structure  (WBS)  and  includes  an  integrated  schedule  and 
budget  based  on  the  project  WBS. 

•  Earned  Value  Management  System  (EVMS)  -  the  process,  procedures,  tools  and 
templates  used  by  an  organization  to  do  earned  value  management. 

As  one  can  see  from  these  definitions,  each  of  these  terms  is  different.  However,  each  is 
dependent  on  the  other. 

For  earned  value  to  work  properly  within  any  organization  there  must  be  five 
basic  data  elements.  They  are  as  follows;  Budgeted  Cost  of  Work  Scheduled  (BCWS)  or 
Planned  Value  (PV),  Budgeted  Cost  of  Work  Performed  (BCWP)  or  Earned  Value  (EV), 
Actual  Cost  of  Work  Performed  (ACWP)  or  Actual  Cost  (AC),  Budget  at  Complete 
(BAC)  or  Planned  Cost,  and  Estimate  at  Complete  (EAC)  or  Forecasted  Cost.  Table  1 
displays  the  five  core  elements  used  with  earned  value  in  the  DoD  (Air  Force  Cost 
Analysis  Handbook,  2007). 
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Table  1  -  Five  Basic  Elements  for  Earned  Value 


Element 

Title 

Common  T  erminology 

BCWS 

Budgeted  Cost  of  Work 

Scheduled 

Planned  Value  (PV),  Performance 
Measurement  Baseline  (PMB),  plan, 
baseline 

BCWP 

Budgeted  Cost  of  Work 

Performed 

Earned  Value  (EV) 

ACWP 

Actual  Cost  of  Work  Performed 

Actual  Cost  (AC),  actuals 

BAC 

Budget  at  Complete 

Planned  Cost 

EAC / LRE 

Estimate  at  Complete  /  Latest 
Revised  Estimate 

Forecasted  Cost 

Figure  4  illustrates  the  basic  relationship  between  the  cumulative  values  of  the  five  basic 
elements  of  EVM  (DAU  Gold  Card,  2013). 


EAC 

BAG 


Figure  4  -  Relationship  between  EVM  elements 
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As  Figure  4  indicates,  the  ACWP  will  equal  the  EAC  at  the  end  of  the  contract  and  the 
BCWS  and  BCWP  should  both  equal  the  BAC  at  the  end. 

Figure  5  displays  the  relationship  between  PV,  AC  and  EV  (Lukas,  2008): 


Figure  5  -  Relationship  between  EVM  elements 

Figure  5  illustrates  the  status  at  the  time  the  project  was  budgeted  to  have  spent  $50  but 
actually  spent  only  about  $35.  At  first  glance,  this  looks  positive  until  earned  value  is 
also  considered.  At  the  time  of  the  status  date,  the  project  actually  earned  only  $25  worth 
of  value.  With  these  three  data  points,  a  project  manager  can  immediately  calculate  the 
cost  variance  (CV),  schedule  variance  (SV),  cost  performance  index  (CPI),  and  schedule 
performance  variance  (SPI). 

The  cost  variance  indicates  the  difference  between  the  earned  value  and  the  actual 
cost.  If  the  CV  is  less  than  zero,  it  means  costs  are  higher  than  the  value  earned  to  date. 
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If  the  CV  is  greater  than  zero,  it  demonstrates  costs  are  lower  than  the  value  earned  to 
date  (Lukas,  2008). 

CV  = EV -AC 

The  schedule  variance  shows  the  difference  between  the  earned  value  and  the 
planned  or  budgeted  value.  If  the  SV  is  less  than  zero,  it  demonstrates  less  work 
accomplished  than  one  planned  to  date.  If  the  SV  is  greater  than  zero,  it  demonstrates 
more  work  accomplished  that  one  planned  to  date  (Lukas,  2008). 

SV  =  EV- PV 

The  cost  performance  index  shows  a  ratio  comparing  the  earned  value  with  the 
actual  value.  The  CPI  determines  the  value  of  every  dollar  spent  on  a  project.  The  CPI 
shows  the  percentage  of  value  gained  for  every  dollar  invested.  The  CPI  demonstrates 
less  money  earned  on  the  project  than  invested  if  it  is  less  than  one  (not  getting  a  full 
dollar’s  worth  of  work).  The  CPI  demonstrates  more  money  earned  on  the  project  than 
invested  if  it  is  more  than  one  (Air  Force  Cost  Analysis  Handbook,  2007). 

CPI  =  EV/AC 

The  schedule  performance  index  shows  a  ratio  comparing  the  earned  value  with 
the  planned  value.  The  SPI  shows  the  efficiency  of  the  work  accomplishments.  If  the 
SPI  is  less  than  one,  it  show  less  work  has  been  done  than  planned,  while  if  it  is  greater 
than  one  it  shows  more  work  has  been  done  than  planned  (Air  Force  Cost  Analysis 
Handbook,  2007). 

SPI  =  EV  /  PV 

Figure  6  displays  how  the  CV,  SV,  CPI,  and  SPI  relate  with  the  PV,  AC,  and  EV. 
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Figure  6  -  Earned  Value  Analysis 


The  earned  value  calculations  in  Figure  6  convey  to  us  the  current  state  of  the 
project.  While  this  information  is  important,  one  of  the  main  purposes  of  EVMS  is  to 
predict  the  estimate  at  complete  (EAC)  or  final  cost  of  a  project  from  the  information  we 
have  now.  According  to  Joseph  Lukas,  there  are  three  different  ways  for  calculating  the 
EAC. 

EACi  assumes  the  CPI  will  remain  1.0  for  the  rest  of  the  project.  This  method 
assumes  that  even  if  the  project  has  been  running  behind  schedule  it  will  automatically 
correct  itself  for  the  remainder  of  the  contract.  This  formula  produces  the  most  optimistic 
outcome  for  a  project. 

EACi  =  AC  +  (BAC-EV) 

EAC2  is  entitled  the  CPI  forecast.  This  formula  assumes  past  cost  performance  is 
the  only  indicator  for  future  performance.  It  assumes  the  CPI  will  remain  constant  during 
the  rest  of  the  project. 

EACi  =  BAC/CPI 
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EAC3  is  entitled  the  CPI  *  SPI  Forecast.  Considered  the  most  pessimistic  or 
worst  case,  this  formula  assumes  past  cost  and  schedule  performance  are  indicators  for 
future  performance  (Lukas,  2008). 

EACs  =  BAC/(CPI  x  SPI) 

The  Defense  Acquisitions  University  Earned  Value  Management  Gold  Card  gives 
two  different  ways  to  calculate  the  EAC,  the  EACcpiandthe  EACcomposite.  The  formula 
below,  demonstrates  how  the  EACcpiis  calculated  (DAU  Gold  Card,  2013): 

EAC cp i  =  ACWP  +  [BAC  -  BCWP]/  CPI 
or 

EACcpi  =  AC  +  [BAC  -  EV]  /  CPI  shows 

The  EACcomposite  is  similar  to  Lukas ’s  EAC3.  The  below  formula  demonstrates  how  the 
EACcomposite  is  calculated. 

EACcomposite  =  ACWP  +  [BAC  -  BCWP]/ [CPI  *  SPI] 

or 

EACcomposite  =  AC  +  [BAC  -  EV]  /  [CPI  *  SPI] 

Figure  7  displays  how  the  EAC  is  calculated. 
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Figure  7  -  Estimate  at  Complete 

The  EAC  at  complete  is  the  summation  of  the  ACWP,  or  AC,  and  the  Estimate  to 
complete  (ETC). 

Another  formula  that  is  widely  used,  when  analyzing  EVM  data,  is  percent 
complete.  The  final  budgeted  amount  compared  to  where  the  project  is  demonstrates  the 
percent  complete.  However,  the  literature  shows  percent  complete  calculated  two 
different  ways.  The  most  commonly  used  formula  is  by  dividing  the  BCWP  by  the  BAC 
(Thickstun,  2010;  DAU,  2013;  Christensen  and  Templin,  2002).  The  other  method  we 
have  seen  percent  complete  calculated  is  by  dividing  ACWP  by  the  BAC  (Trahan,  2009). 
Table  2  illustrates  the  two  different  formulas  and  the  definition  for  each  acronym. 
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Table  2  -  Percent  Complete  Formulas 


%  Complete  = 

BCWP 

BAC 

ACWP: 

Actual  Cost  of  Worked  Performed 

BCWP: 

Budgeted  Cost  of  Worked  Performed 

%  Complete  = 

ACWP 

BAC: 

Budget  at  Completion 

BAC 

History  of  Earned  Value  in  the  DoD 

Earned  value  has  four  major  evolutions  within  the  DoD  over  the  past  six  decades. 
The  DoD  first  started  to  use  a  form  of  earned  value  back  in  1957  with  the  Navy’s  Polaris 
missile  project,  entitled  program  evaluation  and  review  technique  (PERT).  The  original 
purpose  of  PERT  was  to  help  management  create  a  network  model  showing  the  logical 
steps  to  complete  a  project  while  focusing  on  time  and  the  probability  of  success  of  the 
project.  In  less  than  five  years,  this  method  needed  replacing  (Fleming  and  Koppelman, 
1998). 


In  1962,  the  U.S.  Air  Force  expanded  on  the  original  PERT  by  adding  resources 
to  the  network  model,  entitling  the  new  technique  PERT/Cost.  The  evaluation  of  changes 
in  a  project  was  the  most  important  achievement  of  PERT/Cost.  Until  this  time,  the 
evaluation  process  was  comparing  cost  plans  with  the  actual  cost  for  each  project.  With 
PERT/Cost,  project  managers  compared  the  value  of  the  work  to  the  actual  cost.  Today, 
we  look  at  this  as  a  simple  idea,  but  in  the  1960s,  this  changed  the  process  of  project 
evaluations.  Similar  to  PERT,  PERT/Cost  lasted  only  three  years  (Fleming  and 
Koppelman,  1998). 

Even  though  the  DoD  decided  to  stop  using  PERT/Cost,  it  still  believed  earned 
value  concept  had  merit  and  decided  to  retain  earned  value  as  a  tool  to  evaluate  a  project. 
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In  1967,  the  DoD  formally  implemented  earned  value  with  Cost/Schedule  Control 
System  Criteria  (C/SCSC).  Based  on  the  best  practices  from  private  industry  since  the 
early  1900s,  C/SCSC  required  defense  contractors  to  provide  a  minimum  standard  for 
reporting  program  performance,  as  well  as  requiring  contractors  to  integrate  their  cost, 
schedule  and  effort  into  a  single  master  plan.  C/SCSC  was  a  major  improvement  with 
earned  value.  Managers  finally  had  the  resources  to  predict  the  final  total  cost  and  project 
length  more  accurately  (GAO-09-3SP,  2009). 

Cost/Schedule  control  system  criteria  were  DoD’s  first  major  use  of  earned  value 
to  evaluate  projects.  However,  there  were  many  problems  with  the  system.  One  of  the 
biggest  problems  was  the  rigid  requirements.  This  issue  caused  some  contractors  to 
maintain  two  sets  of  data  for  a  project.  One  set  would  be  the  company’s  working  data 
used  to  manage  the  project.  To  meet  the  DoD’s  requirements,  contractors  maintained  a 
second  set  of  data  (GAO,  2009).  Another  issue  contractors  had  with  C/SCSC  was  the 
contractor  thought  that  C/SCSC  did  not  add  value  to  their  projects  by  requiring  them  to 
complete  all  the  requirements  for  the  DoD  (Fleming  and  Koppelman,  1998). 

At  the  end  of  1996,  the  DoD  decided  to  discontinue  C/SCSC  due  to  the  problems 
C/SCSC  was  causing  with  the  contractors.  To  create  an  improved  earned  value  system, 
the  DoD,  along  with  private  industry,  created  the  Earned  Value  Management  System 
(EVMS).  When  the  DoD  implemented  EVMS,  in  1997,  the  DoD  stated,  “they  brought 
EVM  back  to  its  intended  purpose  of  integrating  cost,  schedule,  and  technical  effort  for 
managers  and  providing  reliable  data  to  decision  makers”  (GAO-09-3SP,  2009). 
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Problems  with  Earned  Value  Management  System  in  the  DoD 

Even  with  the  evolution  of  earned  value  over  the  past  six  decades,  the  lingering 
question  of  the  best  way  to  evaluate  data  still  exists.  Solomon  has  written  multiple  times 
about  his  concern  on  the  proper  way  companies  report  their  EVM  data.  He  states  that  in 
the  current  DoD  system,  a  contractor  could  report  100%  complete  for  EVM;  however,  the 
project  fails  to  meet  the  technical  requirements  (Solomon,  2006;  Solomon,  2013). 

Lukas  also  expresses  concerns  with  how  earned  value  is  calculated.  He  suggests 
there  needs  to  be  a  quantitative  method  to  assess  the  project,  not  a  qualitative  approach. 
Stating  a  qualitative  approach  could  lead  to  team  biases  (Lukas,  2008). 

Etxegoien  expresses  concerns  about  risk  in  incorporated  EVM  values.  Stating 
that  risk,  which  is  initially  added  to  the  contract,  is  “locked  in  time  with  the  EVMS 
baseline  while  the  actual  risk  is  measured  and  track  separately  (Etxegoien,  2002).” 

Bushey  and  Etxegoien  both  also  state  that  the  current  EVMS  is  too  restrictive  for 
program  managers.  In  both  of  their  researches,  they  mention  program  managers  need  for 
more  freedom  to  choose  what  level  to  report  elements.  They  also  mention  the  current 
negative  connotation  of  rebase  lining  hinders  program  managers  from  admitting  a 
program  might  be  going  over  budget.  They  mention  this  fear  could  lead  program 
managers  to  misrepresent  the  actual  status  of  a  program  (Bushey,  2007;  Extxegoien, 
2002). 

Relevant  Research  Completed 

Quantitative  research  about  cost  growth  into  lower  level  EVM  data  is  very 
limited.  Most  research  using  EVM  data  looks  only  at  level  one.  We  are  attempting  to 
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determine  the  most  beneficial  level  for  the  DoD  to  calculate  its  EVM  data.  We  have 
found  only  one  study  that  used  quantitative  data  to  evaluate  lower  level  EVM  data.  The 
limited  amount  of  research  in  this  area  is  because  there  is  not  a  lot  of  EVM  data  that  goes 
down  that  low.  However,  we  did  find  multiple  qualitative  studies  supporting  lower  level 
EVM  data  is  more  predictive  and  should  be  used  (DAU  Website,  2013).  However,  for 
the  DoD,  we  require  that  the  contractor  only  report  EVM  data  at  level  three  of  the  WBS 
(MIL-STD-881C,  2011). 

Rosado  (201 1)  conducted  the  only  quantitative  study  on  lower  level  EVM  data 
found  in  our  literature  review.  Rosado  had  two  main  goals  for  his  research.  The  usage  as 
an  early  warning  for  cost  growth  within  a  contract  was  the  first  intention  in  the  creation 
of  a  regression  model  using  lower  level  WBS  EVM  data.  His  second  intent  was  to 
determine  which  specific  program  element  contributed  the  most  to  overall  cost  growth. 

Rosado,  however,  used  only  level  three  WBS  elements  in  his  models.  The  reason 
he  did  not  go  deeper  down  the  WBS  tree  was  due  to  lack  of  commonality  of  program 
elements.  He  wanted  to  be  able  to  compare  like  elements  in  the  different  programs  to  see 
how  they  predict  the  overall  cost  growth.  Since  there  is  no  standardization  below  level 
three,  Rosado  was  unable  to  use  lower  level  elements.  He  concluded  his  research  by 
finding  a  strong  relationship  between  level  three  DT&E  elements  and  the  overall  program 
EAC  growth  (Rosado,  2011). 

While  Rosado  conducted  the  only  quantitative  research,  Bushey  (2007)  and 

Etxegoien  (2002)  both  conducted  qualitative  research  on  the  subject.  Etxegoien  looked  at 

how  program  managers  can  better  use  EVM  data.  The  current  standardized  requirements 

constrain  the  program  manager’s  ability  to  track  and  forecast  cost  growth.  At  whatever 
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level  is  necessary  to  give  clarity  to  the  program  manager,  he  recommends  collecting 
EYM  data.  He  also  recommends  collecting  EYM  data  to  the  lowest  level  possible  on 
elements  that  fall  within  the  critical  path  of  the  program.  Etxegoien  does  not  use  any 
quantitative  models  to  support  his  argument.  His  logical  argument  gave  a  large  basis  for 
his  recommendations  (Etxegoien,  2002). 

Bushey,  like  Etxegoien,  looks  at  how  program  managers  can  best  use  EVM  data. 
In  his  research,  he  says  the  best  way  to  detect  a  problem  early  is  to  have  visibility  of  the 
EV  data  at  the  smallest  level.  He  states  that  by  drilling  down  to  the  lower  levels,  a 
program  manager  can  easier  identify  root  causes  for  cost  growth  or  schedule  slippage. 

He  also  states  that  the  ability  to  view  the  lower  level  (control  account)  EVM  data  allows 
the  program  manager  to  be  able  to  talk  directly  with  the  control  account  manager  (CAM). 
By  speaking  directly  to  the  CAM,  the  program  manager  will  also  be  able  to  receive  even 
further  insight  of  any  possible  issues  with  the  program. 

He  uses  an  analogy  of  a  car:  “if  our  car  would  not  start,  we  would  focus  on  repairs 
at  the  lower-level  starter  system  and  not  on  analysis  at  the  overall  car  level  or  unrelated 
lower-level  areas  such  as  paint,  tires,  or  structure”  (Bushey,  2007).  However,  in  his 
research,  he  states  the  benefits  of  lower  level  data,  but  fails  to  show  a  quantitative 
supporting  argument  for  his  statement. 

Fonnesbeck  and  Lee  conducted  a  study  in  1987.  In  their  research,  the  authors 

were  looking  at  the  WBS  from  a  cost  estimator’s  perspective.  They  wanted  to  be  able  to 

deliver  the  best  cost  estimate  with  the  cost  data  available.  One  of  the  problems  they 

stated  was  the  lack  of  standardization  of  cost  data  below  level  three.  The  main  reason 

they  state  this  requirement  will  not  change  is: 
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the  Office  of  Management  and  Budget  (OMB)  has  established  guidelines 
requiring  OSD  to  minimize  the  reporting  requirements  placed  on  the  contractors 
by  the  CCDR  system.  This  seems  to  concur  with  the  administration  policy  to 
reduce  the  volume  of  bureaucratic  paperwork. 

The  authors  interviewed  different  DoD  contractors  while  conducting  this  study.  The 

authors’  intent  was  to  determine  if  the  contractors  could  provide  cost  data  lower  than 

level  three  of  the  WBS.  They  found  most  contractors  had  cost  data  at  least  one  level 

below  the  PWBS  and  some  had  data  as  low  as  levels  eight  and  nine.  Fonnesbeck  and  Lee 

gave  two  suggestions  on  how  to  improve  cost  report,  even  if  we  cannot  change  the 

requirement  for  contractors  to  only  report  down  to  level  three.  One  of  their  suggestions 

dealt  with  creating  a  time-phased  data  reporting  system  and  a  data  base  system  to  store 

EVM  data.  The  other  suggestion,  however,  is  to  remove  some  level  three  elements  and 

replace  them  with  their  level  four  sub-elements.  This  sounds  simple,  but  the  authors 

found  in  many  projects  that  the  level  four  element  could  account  for  more  than  40%  of 

the  level  two  element  above  it.  By  not  being  able  to  track  the  level  four  element  closely, 

it  makes  reporting  where  a  problem  is  more  difficult,  if  not  impossible  (Fonnesbeck  and 

Lee,  1987). 

Conclusion 

In  the  chapter,  we  provided  a  brief  look  into  how  a  WBS  works  and  the 
relationships  between  the  CWBS  and  the  PWBS.  We  also  explained  the  definition  of 
earned  value  with  some  basic  formulas  and  definitions.  Then,  we  went  over  the  history 
and  evolution  of  earned  value  in  the  DoD.  Through  our  research,  we  have  not  seen  any 
research  looking  into  the  exact  problem  we  are  researching.  However,  Rosado, 
Etxegoien,  Bushey,  Fonnesbeck  and  Lee  demonstrate  there  have  been  some  studies 
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completed  on  this  topic,  but  these  authors  lacked  the  ability  to  obtain  cost  data  down  to 


the  control  account  level.  In  the  next  chapter,  we  will  explain  our  data  collection  process 
and  methodology  for  evaluating  our  cost  data  in  detail. 
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Chapter  3  -  Data  Collection  and  Methodology 


Chapter  Overview 

This  chapter  describes  our  data  collection  and  methodology  in  our  research.  The 
objective  of  this  chapter  is  to  make  clear  the  steps  we  performed  in  our  analysis  and  data 
collection  and  give  the  reason  behind  our  choices.  We  also  briefly  explain  other 
techniques  we  attempted,  as  well  as  the  analysis  and  the  reason  we  chose  not  to  use  them. 
By  the  end  of  this  chapter,  we  will  have  shown  you  how  we  collected  our  data  and  how 
we  limited  it.  We  will  also  have  shown  our  criteria  for  evaluating  our  research  question 
and  the  steps  we  took  to  obtain  our  results. 

Defense  Cost  and  Resource  Center  (DCARC) 

History  and  Intent 

Established  in  1998,  DCARC’s  primary  role  is  to  collect  past  and  present  Major 
Defense  Acquisition  Programs  in  one  central  location.  DCARC  has  four  main  objectives. 
The  first  is  to  allow  one  location  to  collect  contractor’s  cost  and  software  data  reports. 

The  second  allows  authorized  users  access  to  this  cost  data  for  analysis  for  future 
projects.  The  third  objective  is  to  maintain  the  integrity  and  accuracy  of  the  data 
collected  from  contractors.  The  final  object  is  to  improve  the  quality  of  data  reported  by 
industry.  The  overall  goal  of  DCARC  is  to  provide  senior  leaders  with  accurate  and 
timely  cost  estimates  in  order  to  provide  the  war  fighter  the  weapons  and  equipment 
needed  to  win  (DCARC,  2007). 
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Earned  Value  Management  Central  Repository 

We  collected  the  data  for  our  research  from  the  Earned  Value  Management 
Central  Repository  (EVM-CR)  found  on  the  Defense  Cost  and  Resource  Center 
(DCARC)  portal.  Designating  a  single  place  where  contractors  can  submit  their  Contract 
Performance  Reports  (CPRs),  Contract  Funds  Status  Reports  (CFSRs),  and  Integrated 
Master  Schedules  (IMSs)  is  the  function  of  the  EVM-CR  (Office  of  the  Secretary  of 
Defense  Central  Repository  for  Earned  Value  Management  (EVM)  Data  Manual,  2008). 
Performance  Assessments  and  Root  Cause  Analysis  (PARCA)  directly  oversees  EVM- 
CR. 

A  contractor  must  submit  seven  different  CPRs.  Contractors  have  used  the  first 
five  formats  for  years.  However,  in  2012,  formats  six  and  seven  were  established 
(PARCA  Memo,  2012).  The  intent  of  these  two  new  formats  is  to  better  integrate  cost 
and  schedule  reporting.  Format  six  contains  the  contract’s  Integrated  Master  Schedule 
(IMS)  and  format  seven  contains  an  electronic  history  and  forecast  file.  The  format  we 
will  focus  on  in  our  research  is  format  seven. 

The  EVM-CR  sorts  all  of  the  programs  into  five  different  organizations.  The  five 
organizations  are  Army,  Air  Force,  Navy,  Department  of  Defense,  and  Missile  Defense 
Agency  (MDA).  There  are  currently  143  different  programs  and  422  contracts  found  on 
the  EVM-CR.  Table  3  displays  the  breakdown  by  organization. 
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Table  3  -  EVM-CR  Program  Breakdown 


#  of  Programs 

#  of  Contracts 

Army 

34 

74 

Air  Force 

40 

93 

Navy 

46 

181 

DOD 

12 

50 

MDA 

11 

24 

Total 

143 

422 

Data  Collection 

In  this  section,  we  will  explain  the  process  and  steps  required  to  create  our  data 
set  and  subsequently  use  in  our  evaluation  of  our  research  questions.  The  first  step 
implemented  was  to  request  access  to  the  EVM-CR  database  through  PARCA  for 
approval.  Initially,  we  only  had  access  granted  to  the  Army’s  EVM  data.  This  restriction 
was  due  to  policies  at  PARCA  limiting  access  only  to  the  service  of  the  requesters.  In 
order  to  gain  access  to  the  Navy’s  and  the  Air  Forces’  EVM  data,  we  had  to  submit  an 
exception  to  the  policy  and  then  each  branch’s  representative  had  to  approve  our  request. 
Ultimately,  the  Army,  Air  Force  and  Navy  cost  data  were  the  only  agencies  we  had 
permission  to  access.  Both  the  Missile  Defense  Agency  and  DoD  projects  did  not  grant 
access.  However,  we  felt  the  three  services  we  had  would  provide  sufficient  data  to 
conduct  our  research. 

Selecting  Contract  Data 

Once  we  had  access  to  the  EVM  data,  we  determined  the  criteria  we  needed  in 
order  to  answer  our  research  questions.  We  looked  through  every  contract  (to  date)  for 
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the  Army,  Air  Force  and  Navy.  The  first  and  most  important  requirement  was  that  the 
contract  had  to  have  EYM  data  down  to  at  least  level  five  of  the  WBS.  We  also  needed 
to  ensure  that  the  level  five  EYM  data  was  different  from  the  level  three  data.  In  many  of 
the  contracts,  we  found  level  three  and  five  data  identical.  In  these  cases,  we  removed 
those  elements  from  our  analysis. 

Our  second  criterion  was  that  each  contract  has  at  least  twelve  consecutive 
periods  of  EVM  data.  The  reason  for  the  requirement  was  to  allow  us  to  analyze  our 
explanatory  variable  over  a  length  of  time.  We  determined  any  contract  with  less  than 
twelve  periods  of  EVM  data  would  limit  the  predictive  capabilities  of  our  model. 

Our  final  criterion  was  that  the  data  was  complete  and  relevant.  The  system  was 
able  to  tell  us  if  the  contractor  had  lower  level  data  and  if  the  contract  was  over  twelve 
months,  however,  the  system  could  not  tell  us  if  the  data  was  complete  or  if  all  the 
months  had  EVM  data.  In  our  screening  of  these  criteria,  we  found  that  many  elements 
reported  by  the  contractor  failed  to  have  complete  EVM  data.  Many  times  contractors 
would  only  report  their  Latest  Revised  Estimate  (LRE).  We  also  found  in  some 
contracts,  the  contractor  left  multiple  months  of  EVM  data  empty. 

Contract  Selection  Criteria 

After  we  determined  the  criteria  for  our  data,  we  needed  to  create  requirements 
for  the  individual  contract  in  the  EVM-CR.  The  EVM-CR  database  organizes  its  EVM 
data  into  six  different  types  of  files.  They  are  Integrated  Program  Management  Report 
(IPMR)  Cost,  Formatted  Cost,  IPMR  Schedule,  Native  Schedule,  Contract  Fund  Status 
Report  (CFSR),  and  History  files  (Format  7).  However,  we  found  the  best  way  to  obtain 
our  data  was  by  looking  only  for  history  files.  The  history  file  presented  each  of  the 
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WBS  elements  and  their  cumulative  changes  over  the  length  of  the  project.  Contracts  are 
required  to  upload  a  new  history  file  to  the  EYM-CR  at  least  annually.  In  cases  where 
there  were  more  than  one  history  file  with  a  contract,  we  only  downloaded  the  most 
recent  published  history  file. 

In  order  to  view  and  sort  the  history  files  we  downloaded,  we  used  the  CPR  File 
Viewer  software  provided  from  the  DCARC  website.  This  software  allowed  us  to 
quickly  sort  and  evaluate  the  different  contracts.  Once  we  sorted  the  contracts,  we  used 
the  software  to  download  the  files  into  Microsoft  Excel  format.  We  then  used  these 
Excel  files  to  define  our  explanatory  variables  to  use  in  our  statistical  analysis. 

Overview  of  Data 

After  applying  our  data  and  contract  criteria,  we  narrowed  the  number  of 
contracts  from  422  to  40.  Appendix  B  contains  a  list  of  the  forty  contracts.  Table  4 
illustrates  the  breakdown  by  criteria.  Column  1  and  2  indicate  the  number  of  programs 
and  contracts  in  each  service.  The  following  four  columns  are  the  criteria  for  limiting 
data.  The  numbers  in  Table  4  are  dependent  on  the  number  to  the  left.  An  example,  the 
Army  had  thirty  contracts  with  at  least  level  five  data  and  only  twenty-five  of  them  were 
greater  than  twelve  months.  The  final  column  calculates  the  percentage  of  contracts  with 
usable  data. 
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Table  4  -  Data  Overview  by  Criteria 


#  of 

Programs 

#  of 

Contracts 

#  with  History 
Files 

Lvl  5  or  greater 

WBS 

>12  Months 

Complete 

Data 

%  Complete 

Army 

34 

74 

52 

30 

25 

9 

12.16% 

Air  Force 

40 

93 

54 

24 

18 

4 

4.30% 

Navy 

46 

181 

111 

44 

30 

27 

14.92% 

DOD 

12 

50 

21 

ACCESS  NOT  GRANTED 

MDA 

11 

24 

6 

Total 

143 

422 

244 

98 

73 

|  40 

9.48%  | 

From  looking  at  Table  4,  we  can  identify  the  breakdown  of  the  forty  contracts  by 
department.  We  also  see  a  majority  of  the  contracts  came  from  Navy  contracts. 
Furthermore,  Table  4  makes  evident  the  Air  Force  has  the  smallest  percentage  of  usable 
contracts. 

Observing  just  the  forty  contracts  used  in  our  analysis,  we  broke  the  contracts 
down  by  which  phase  the  contract  was  in  (RDT&E,  Production,  Other  or  Unknown), 
ACAT  level  (IAC,  IAM,  IC,  ID,  II,  III  or  Unknown),  contract  type  (CPAF,  CPFF,  CPIF, 
FFP,  or  IDIQ)  and  type  of  WBS  (Aircraft,  Electronic/ Automated  Software,  Missile, 
Ordnance,  Ship,  Space,  UAV  or  Other).  Table  5  illustrates  the  breakdown  by  phase  of 
contract. 


Table  5  -  Data  by  Phase 


#  of 

Programs 

#  of 

Contracts 

#with  History 
Files 

SDD 

Prod 

Other 

Service 

Unknown 

34 

74 

46 

5 

3 

0 

0 

1 

40 

93 

49 

3 

1 

0 

0 

0 

46 

181 

104 

14 

9 

1 

0 

3 

DOD 

12 

50 

16 

ACCESS  NOT  GRANTED 

11 

24  ! 

3 

Total 

143 

422  ; 

218 

22 

13 

1 

0 

4 
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As  Table  5  displays,  all  but  five  of  the  contracts  are  either  RDT&E  or  Productions 
contracts. 

Table  6  illustrates  the  breakdown  by  ACAT. 


Table  6  -  Data  by  ACAT 


#  of  Programs 

#  of  Contracts 

Complete 

Data 

IAC 

1AM 

1C 

ID 

II 

III 

Unknown 

Army 

34 

74 

9 

0 

0 

3 

5 

0 

0 

1 

Air  Force 

40 

93 

4 

1 

1 

0 

1 

0 

1 

0 

Navy 

46 

181 

27 

0 

0 

12 

15 

0 

0 

0 

DOD 

12 

50 

ACCESS  NOT  GRANTED 

MDA 

11 

24 

Total 

143 

422 

40 

1 

_ 

1 

15 

21 

0 

1 

Table  7  indicates  the  breakdown  by  contract  type 


Table  7  -  Data  by  Contract  Type 


#of  Programs 

#  of  Contracts 

Complete 

Data 

CPAF 

CPFF 

CPIF 

FFP 

FPIF 

IDIQ 

Army 

34 

74 

9  ! 

1 

1 

4 

2 

1 

0 

Air  Force 

40 

93 

4 

1 

1 

0 

1 

0 

1 

Navy 

46 

181 

27 

11 

5 

6 

0 

4 

1 

DOD 

12 

50 

ACCESS  NOT  GRANTED 

MDA 

11 

24 

Total 

143 

422 

40  | 

13 

7 

10 

3 

5 

2 

Table  8  illustrates  the  data  by  type  of  WBS. 
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Table  8  -  Data  by  type  of  WBS 


#  of  Programs 

#  of  Contracts 

Complete 

Data 

Aircraft 

ELECTRONIC/ 

AUTOMATED 

SOFTWARE 

Missile 

Ordance 

Ship 

Space 

UAV 

Other 

Army 

34 

74 

9 

0 

4 

1 

2 

0 

0 

0 

2 

Air  Force 

40 

93 

4 

1 

2 

0 

0 

0 

1 

0 

0 

Navy 

46 

181 

27 

10 

5 

4 

0 

5 

0 

3 

0 

DOD 

12 

50 

ACCESS  NOT  GRANTED 

MDA 

11 

24 

Total 

143 

422 

40 

11  11  5  2  5  1  3  2 

Table  9  lists  each  of  the  forty  contracts  and  the  total  number  of  level  five  elements 


in  the  contract.  Table  9  also  breaks  down  the  size  of  the  level  five  elements  as  a 


percentage  of  the  level  one  EAC. 
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Table  9  -  Total  number  of  level  5  Elements 


Total  #  of  Ivl  5 

#  Ivl  5  w/  EVM  data 

>.25% 

ON 

O 

lO 

A 

>1% 

B2  MOP  -  Massive  Ordance  Penetrator  (F33657-99-D-0028) 

126 

66 

17 

9 

7 

Chem  Demil  -  CMA  (DAAA09-97-C-0025) 

2494 

1520 

63 

19 

9 

Chem  Demil  -  CMA  (DACA87-89-C-0076) 

441 

293 

41 

30 

20 

Excalibur  (DAAE30-98-C-1032) 

23 

21 

20 

18 

15 

Excalibur  (W15QKN-08-C-0530) 

22 

20 

20 

20 

16 

FBCB2  (W15P7T-04-D-G205) 

443 

300 

96 

53 

18 

IAMD  (W31P4Q-08-C-0418) 

495 

345 

83 

48 

19 

ISPAN  ( FA8722-04-C-0009) 

141 

104 

65 

40 

29 

JAGM  (W31P4Q-08-C-A123) 

34 

33 

25 

23 

21 

JTRS  ( D  A  A  B 07-02-  C-  C403 ) 

29 

24 

7 

4 

1 

MPS  -  SEIC  ( FA8720-05-C-0005) 

54 

28 

25 

23 

15 

NAVSTAR  GPS  (FA8807-06-C-0001) 

302 

160 

31 

14 

10 

WGS  ( FA8808-06-C-0001) 

54 

21 

20 

20 

15 

WIN-T  INC3  (DAAB07-02-C-F404) 

75 

68 

37 

31 

15 

AAG  -  Advanced  Arresting  Gear  Program  (N68335-03-C-0205) 

608 

419 

101 

60 

19 

AGM-88E  (Advanced  Anti-Radiation  Guided  Missile  (N00019-03-C-0353) 

96 

78 

39 

27 

17 

AIM-9X  Block  II  ( N 00019-  12-C-2002) 

48 

25 

22 

16 

14 

AMDR  -  Air  &  Missile  Defense  Radar  ( N 00024- 10- C- 5359) 

33 

17 

13 

13 

10 

CEC  -  Cooperative  Engagement  Capability  (N00024-05-C-5100) 

104 

73 

45 

33 

22 

DDG  1000 -Zumwalt  Class  Destroyer  (N00024-05-C-5346) 

3820 

2746 

69 

31 

7 

DDG51  -  Arleigh  Burke  Class  Guided  Missile  Destroyer  (N00024-02-C-2304) 

27 

21 

16 

15 

12 

EA-18G  -  Airborne  Electronic  Attack  variant  (N00019-04-C-0005) 

10 

7 

3 

3 

2 

HI  Upgrades  (N00019-06-G-0001) 

28 

23 

12 

12 

6 

JPALS  -  Joint  Precision  Approach  and  Landing  System  (N00019-08-C-0034) 

178 

109 

36 

28 

15 

JPALS  -  Joint  Precision  Approach  and  Landing  System  ( N 000 19- 08- C- 0034)  (2) 

49 

41 

32 

21 

15 

JSOW  (N00019-05-G-008-DO) 

12 

12 

10 

10 

9 

LCS  -  Littoral  Combat  Ship  (N00024-03-C-2310) 

1729 

524 

38 

28 

21 

LCS  -  Littoral  Combat  Ship  ( N 00024- 11-C- 2301) 

87 

32 

18 

17 

15 

LPD  17-  San  Antonip  Class  ( N 00024- 04- C- 2204) 

2193 

308 

47 

27 

14 

MH-60R  (N00019-04-C-0130) 

50 

47 

33 

28 

22 

MH-60R  ( N 00019-08- C- 0005 ) 

53 

48 

32 

27 

23 

MH-60R  ( N00019-09-C-0059) 

24 

18 

12 

12 

10 

MH-60S  ( N  00019-03-C-0003) 

101 

80 

43 

33 

17 

MQ-4C  Triton  (N00019-08-C-0023) 

772 

523 

51 

24 

11 

P-8A  Poseidon  Program  ( N 000 19- 04- C- 3 146) 

38 

24 

14 

12 

9 

SM-6 Standard  Millile  6  ( N 00024- 04- C- 5344) 

104 

100 

61 

37 

22 

SM-6 Standard  Millile  6  (N00024-09-C-5305) 

81 

77 

37 

24 

18 

V- 22  -  Osprey  Joint  Advanced  Vertical  Aircraft  (N 00421- 10- D-0012) 

4 

4 

4 

4 

3 

V- 22-  Osprey  Joint  Advanced  Vertical  Aircraft  (N 61339-08- D-0004) 

36 

23 

10 

7 

5 

V-22-  Osprey  Joint  Advanced  Vertical  Aircraft  (N 61340- ll-C-0004) 

17 

11 

9 

8 

7 

VTUAV  ( N00019-00-C-0277) 

204 

157 

52 

29 

16 

VTUAV  ( N  000 19- 12-  C-0059) 

725 

429 

58 

25 

10 

Number  of  Level  5  elements 

15964 

8979 

1467 

963 

581 

Percent  of  total  #  level  5 

56.25% 

9.19% 

6.03% 

3.64% 
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Table  9  reveals  that  56.25%  of  all  the  level  five  elements  actually  had  EYM  data.  In 
addition,  we  can  see  that  less  than  10%  of  all  the  level  five  elements  are  larger  than 
0.25%  of  the  total  contract. 

Contract  Error 

While  going  through  the  forty  contracts,  we  found  numerous  errors  within  the 
contracts.  Although  these  errors  did  not  seem  intentional,  we  needed  to  address  the  errors 
the  best  we  could.  We  either  removed  the  WBS  element  from  the  calculations  or 
attempted  to  reconstruct  the  intended  value.  Some  examples  of  attempting  to  reconstruct 
the  intended  value  are  when  the  WBS  element  is  on  a  constant  growth  and  then  a  month 
is  blank.  After  which,  the  WBS  element  continues  to  grow  at  the  previous  growth  rate. 
Table  10  notionally  demonstrates  this  error. 


Table  10  -  Notional  Example  #1  of  Contract  Errors 


Time 

1 

2 

3 

4 

5 

6 

WBS  Element 

$5 

$10 

$15 

$25 

$30 

In  this  insistence,  we  would  extrapolate  the  EYM  values  for  the  missing  month.  In  this 
example,  we  would  have  inserted  $20  in  the  fourth  month.  Table  1 1  illustrates  another 
common  error  we  encountered.  In  this  type  of  error,  a  contractor  inserted  a  monthly 
change  instead  of  the  cumulative  change  for  a  WBS  element. 


Table  11  -  Notional  Example  #2  of  Contract  Error 


Time 

1 

2 

3 

4 

5 

6 

WBS  Element 

$10 

$20 

$30 

$10 

$50 

$60 
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In  this  insistence,  we  would  add  the  monthly  change  value  to  the  last  reported  cumulative 
value.  In  this  example,  we  would  have  inserted  $40  in  the  fourth  month.  ACWP, 

BCWP,  and  BCWS  were  common  places  we  found  this  type  of  error  because  of  the 
constant  growth  of  these  elements.  We  found  212  errors  in  twenty  of  our  forty  contracts. 
Table  12  details  the  twenty  contracts  with  errors  and  the  number  of  errors  in  each 
contract. 


Table  12  -  Number  of  Errors  per  Contract 


Service 

Project  Name 

Prime  Contract  Number 

#of 

errors 

within 

contract 

Level  3 

#of 

errors 

within 

contract 

Level  5 

Air  Force 

B-2  MOP  -  Massive  Ordnance  Penetrator 

F 33657- 99-  D- 0028 

0 

1 

Navy 

CEC- Cooperative  Engagement  Capability 

N 00024- 05-  C-  5 100 

1 

50 

Army 

Chem  Demil  -  CMA 

DACA87-89-C-0076 

0 

1 

Army 

Chem  Demil  -  CMA 

D  A  A  A09- 97-  C- 0025 

1 

2 

Navy 

DDG  1000  -  ZUMWALT  CLASS  Destroyer 

N 00024- 05-  C- 5346 

0 

1 

Army 

IAMD  -  Integrated  Air  &  Missile  Defense 

W31P4Q-08-C-0418 

0 

2 

Navy 

- 1 

LCS  -  Littoral  Combat  Ship 

N 00024- 1 1-  C-  2301 

20 

0 

Navy 

LPD  17  -  SAN  ANTONIO  CLASS  Amphibious  Transport 
Dock  Ship 

N 00024- 04-  C-  2204 

0 

6 

Navy 

MH-60R  -  Multi-Mission  Helicopter  Upgrade 

N 000 19-09-  C- 005 9 

0 

3 

Navy 

MH-60S  -  Multi-Mission  Combat  Support  Helicopter 

N 000 19- 03-  G- 0003 

0 

10 

Navy 

MQ-4C  Triton  (Formerly  BAMS) 

N 000 19- 08-  C- 0023 

0 

12 

Navy 

P-8A  -  Poseidon  Program 

N 000 19- 04-  C-  3 146 

0 

11 

Navy 

SM-6-Standard  Missile-6 

N 00024- 04-  C- 5344 

0 

5 

Navy 

V-22  -  OSPREY  Joint  Advanced  Vertical  Lift  Aircraft 

N 00421- 10-  D-0012 

0 

1 

Navy 

V-22  -  OSPREY  Joint  Advanced  Vertical  Lift  Aircraft 

N  61340- 11- C- 0004 

0 

7 

Navy 

V-22  -  OSPREY  Joint  Advanced  Vertical  Lift  Aircraft 

N  61339-08- D-0004 

1 

1 

Navy 

VTUAV  -  Vertical  Takeoff  and  Land  Tactical  Unmanned 
Air  Vehicle  (Fire  Scout) 

N 000 19- 00-  C- 0277 

0 
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Navy 

VTUAV  -  Vertical  Takeoff  and  Land  Tactical  Unmanned 
Air  Vehicle  (Fire  Scout) 

N00019-12-C-0059 

0 

26 

Air  Force 

WGS- Wideband  Global  SATCOM  Program 

FA8808-06-C-0001 

0 

3 

Army 

WIN-T  Inc.  3  -  Warfighter  Information  Network 
Tactical  Increment  3 

DA  A  B07- 02-  C-  F  404 

0 

1 
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Table  12  indicates  89%  of  the  errors  we  detected  were  in  level  five  elements. 

Methodology 

EAC  Growth 

We  decided  to  use  the  EAC  growth  as  our  predictive  variable.  The  main  reason 
we  decided  to  use  EAC  as  our  predictive  variable  is  because  EAC  allows  us  to  project 
where  the  contract’s  final  cost  is  heading.  The  issue  we  found  with  using  the  EAC  is 
what  value  do  we  use?  For  each  element,  the  contractor  provides  an  EAC.  However,  the 
usage  of  the  formulas  stated  in  Chapter  Two  cannot  calculate  the  contractor’s  EAC.  In 
most  cases,  contractors  provide  their  latest  revised  estimate  (LRE)  as  the  EAC  instead  of 
using  an  established  formula.  The  contractor’s  LRE  is  their  estimation  of  future  cost  for 
the  element.  It  can  include  factors  outside  of  EYM  data.  This  caused  an  issue  with  our 
research  in  determining  the  predictive  capabilities  of  the  lower  level  earned  value  data. 

In  the  next  section,  we  will  discuss  the  way  we  decided  to  calculate  the  EAC.  As  stated 
in  Chapter  Two,  there  are  many  different  ways  EAC  can  be  calculated  using  earned  value 
data. 

Defining  Percent  EAC  Growth 

First,  we  had  to  determine  how  we  would  calculate  our  EAC.  As  stated  earlier, 
the  EAC  provided  by  the  contractor  in  the  EVM  data  was  actually  the  contractor’s  LRE. 
However,  the  LRE  is  very  subjective  and  we  decided  not  to  only  use  this  value  for  our 
EAC.  We  decided  to  also  use  the  composite  EAC  as  discussed  in  Chapter  Two.  The 
reason  we  chose  this  formula  is  that  it  is  the  most  conservative  formula  for  the  EAC. 
Below  is  the  formula  for  the  EACCOmposite: 
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EACcomposite  =  AC  +  [BAC  -  EV]  /  [CPI  *  SPI] 

We  were  able  to  calculate  the  EAC  composite  using  the  EVM  data  provided  by  the 
contractor  from  the  EYM-CR.  We  then  recalculated  all  the  monthly  EACs  for  each 
element  at  level  one,  three  and  five. 

Defining  Lower  Level  EAC  Growth 

We  evaluated  two  different  techniques  to  measure  levels  three  and  five  cost 
growth.  The  first  method  examined  only  the  cost  growth  of  the  individual  elements 
within  each  level.  Definitions  #1  and  #2  used  this  technique.  The  second  technique 
examines  the  overall  cost  growth  within  a  level.  We  used  this  technique  with  definitions 
#3  and  #4.  Additionally,  the  second  technique  is  the  method  Rosado  used  in  his  research. 
Figure  8  displays  each  of  the  four  definitions  and  if  it  used  the  EAC  or  LRE  and  which 
cost  growth  technique  was  used.  The  next  section  explains  each  of  these  definitions 
further. 


EAC 

EAC 

Cum  Change 

Definition  #3 

Element  Change 

Definition  #1 

LRE 

LRE 

Cum  Change 

Element  Change 

Definition  #4 

Definition  #2 

Figure  8  -  EAC  Growth  Definitions 
Explanatory  Variables 

Once  we  recalculated  our  predictive  variable,  EAC  growth,  we  created  our  level 
three  and  five  explanatory  variables  to  use  in  our  different  statistical  models.  We  used 
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the  four  definitions  in  Figure  8  for  both  our  level  three  and  five  variables.  Figure  9 


illustrates  each  definition.  In  all  four  instances,  we  created  a  ratio  by  dividing  the 
monthly  change  of  our  predictive  variable  by  the  last  reported  level  one  EAC.  By 
creating  a  ratio,  in  respects  to  the  final  level  one  EAC,  it  allowed  us  to  normalize  our 
data. 


Figure  9  -  Level  three  and  Five  Explanatory  Variables 
Cost  Growth  Definition  #1 

In  most  cases,  not  all  level  three  or  five  elements  start  at  the  beginning  of  the 
contract.  Our  first  definition  looked  only  at  the  cumulative  change  in  each  level’s  EAC. 
This  method  excluded  the  initial  cost  of  an  element.  Table  13  displays  this  example. 
This  method  allowed  us  to  compare  only  the  cost  growth  of  lower  level  elements. 
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Table  13  -  Cost  Growth  Definition  #1  Example 


Time 

1 

2 

3 

4 

5 

Element 

1 

$10 

$15 

$20 

$25 

$30 

2 

$10 

$10 

$10 

$15 

3 

$50 

$100 

Monthly  Change 

$0 

$5 

$5 

$5 

$60 

Cum  Change 

$0 

$5 

$10 

$15 

$75 

Cost  Growth  Definition  #2 


With  the  exception  of  the  usage  of  the  contractor’s  EAC  provided  in  the  contract, 
the  calculated  variable  used  the  same  technique  as  definition  #1.  This  allowed  us  to 
evaluate  if  there  was  a  difference  using  the  contractor’s  EAC  or  recalculating  the  EAC 
using  an  established  formula. 

Cost  Growth  Definition  #3 


Similar  to  the  first  definition,  this  looked  at  our  recalculated  EAC.  Definition  #3 
differs  from  the  first  by  including  the  cost  of  new  elements  into  our  calculations  of  the 
monthly  cumulative  change.  Table  14  illustrates  this  example. 


Table  14  -  Cost  Growth  Definition  #3  Example 


Time 

1 

2 

3 

4 

5 

1 

$10 

$15 

$20 

$25 

$30 

2 

$10 

$10 

$10 

$15 

3 

$50 

$60 

Monthly  Change 

$10 

$15 

$5 

$55 

$60 

Cum  Change 

$10 

$25 

$30 

$85 

$145 
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Cost  Growth  Definition  #  4 


Definition  #4  is  another  test  variable  using  the  same  technique  as  definition  #3, 
except  the  contractors  EAC.  Again,  for  this  definition,  we  created  it  to  compare  a 
recalculated  EAC  with  the  contractor’s  provided  EAC  (LRE). 

Normalizing  Cost  Growth 

Our  contracts  ranged  in  value  from  $19  million  to  over  five  billion  dollars.  In 
order  to  compare  the  wide  range  of  data,  we  created  a  cost  growth  ratio  for  each  of  our 
eight  explanatory  variables.  Our  ratio  compares  the  cumulative  change  divided  by  the 
final  level  one  EAC. 

Level  3  or  5  Cum  Change 

Cost  Growth  Ratio  = - 

Final  Level  1  EAC  (or  LRE ) 

Normalizing  Contract  Length 

Our  contracts  ranged  from  twelve  months  to  over  eighty  months.  In  order  to 
compare  this  wide  range  of  data,  we  calculated  the  percent  complete  of  level  one  for  each 
month  of  the  contract.  The  formula  we  used  to  calculate  the  percent  complete  was 
(Tracy,  2005): 

Percent  Complete  =  CumBCWS/BAC 

Upon  completion  of  the  percent  complete  calculation  for  each  month,  we  created 
bins  for  every  5%  complete.  A  bin  would  contain  any  month  falling  within  plus  or  minus 
2.5%  of  the  bin  value.  For  example,  the  40%  complete  bin  would  contain  any  month 
with  a  percent  complete  greater  than  or  equal  to  37.5%  and  less  than  42.5%.  In  cases 
where  more  than  one  month  fell  into  a  bin,  we  used  only  the  last  month  that  fall  within 
the  bin. 
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Multiple  Regression 

The  technique  we  used  in  our  analysis  was  Ordinary  Least  Squared  (OLS) 
multiple  regression.  We  used  OLS  multiple  regression  to  create  and  analyze  our  four 
different  definitions  of  cost  growth.  There  were  three  main  reasons  why  we  chose  this 
method.  The  first  reason  for  choosing  OLS  multiple  regression  was  to  evaluate  the 
overall  model  using  the  F-test.  We  then  compared  the  F-test  value  to  our  alpha  (.05). 

The  model  failed  if  the  F-test’s  p-value  was  greater  than  the  alpha.  The  second  reason  we 
chose  this  method  was  that  it  provided  a  T-test.  The  T-test  evaluates  each  individual 
independent  variable.  We  used  Bonferroni  correction  to  determine  if  each  independent 
variable  was  significant.  Bonferroni  correction  compares  the  independent  variable’s  p- 
value  to  our  alpha  (.05)  divided  by  the  number  of  independent  variables  in  the  model.  An 
example  using  Bonferroni  correction  is  if  there  are  five  independent  variables  then  for  the 
variable  to  be  significant,  its  p-value  needs  to  be  less  than  0.01  (0.05  divided  by  five). 

The  final  analysis  this  method  provided  was  a  variance  inflation  factor  (VIF)  score.  The 
VIF  score  quantifies  the  magnitude  of  multicollinearity  of  the  independent  variables  in 
the  regression  model.  In  order  to  determine  if  two  or  more  independent  variables 
correlated,  we  ensured  that  each  variables  VIF  score  was  less  than  five.  Any  value  over 
five  meant  that  the  particular  variable  had  a  higher  than  acceptable  amount  of  correlation 
with  at  least  one  other  variable. 

We  also  used  stepwise  regression  to  determine  which  independent  variables  were 

included  in  our  models.  Stepwise  regression  is  an  automatic  process  conducted  by  our 

statistical  software.  The  software  adds  or  removes  variables  attempting  to  create  the  most 

predictive  model  available  with  the  independent  variables  available.  We  used  stepwise 
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regression  to  determine  our  variables  because  in  each  of  our  models  we  had  our  level 
three  and  five  explanatory  variables  along  with  over  fifty  independent  variables. 

Appendix  C  contains  a  list  of  all  the  independent  variables  used  in  our  models.  Since  we 
were  not  sure  which  variables  would  be  the  most  predictive,  we  used  stepwise  regression 
to  determine  which  ones  to  add. 

The  final  tool  we  used  with  our  multiple  regression  was  Cook’s  Distance  or 
Cook’s  D.  Cook’s  D  allowed  us  to  determine  if  a  certain  contract  was  overly  influential 
in  our  regression  output.  We  defined  a  contract  as  being  overly  influential  if  its  Cook’s  D 
score  was  greater  than  0.5. 

Fit  Y  bv  X 

We  also  evaluated  the  relationship  between  our  level  three  and  five  EYM  data. 

We  used  a  fit  Y  by  X  to  examine  this  relationship.  The  fit  Y  by  X  compares  the 
relationship  between  one  independent  variable  and  the  response  variable.  We  identified 
our  level  three  variable  as  the  response  variable  and  the  level  five  variable  as  our 
explanatory  variable.  This  technique  allowed  us  to  compare  our  level  three  and  our  level 
five  data  to  detect  any  correlation  between  the  level  three  and  five  variables.  We  used 
this  technique  for  definitions  #3  and  #4  at  each  5%  complete  bin.  This  technique  allowed 
us  to  compare  the  slope  of  the  fit  line  to  determine  the  relationship  between  level  three 
and  five.  A  slope  of  one  would  indicate  there  is  perfect  correlation  between  level  three 
and  level  five  data. 

The  software  package  we  used  in  our  analysis  was  JMP®  1 1.0.  There  were  two 

reasons  we  chose  this  software.  First,  JMP®  was  able  to  perform  all  the  different  types  of 

analysis  we  required  and  could  handle  the  amount  of  variables  we  needed  to  include.  The 

43 


other  reason  we  chose  JMP®  was  our  familiarity  with  the  software.  Key  elements  for 
choosing  the  software  were  due  to  prior  knowledge  of  the  software,  wide  availability  and 
its  capabilities. 

Conclusion 

In  this  chapter,  we  described  our  data  collection  methods.  We  also  created 
criteria  for  the  contracts,  as  well  as  for  the  data  within  the  contracts.  We  described  our 
data  set  and  explained  our  methodology  to  answer  our  research  questions.  In  the  next 
chapter,  we  review  the  results  of  our  data  in  the  different  statistical  models  described. 
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Chapter  4  -  Analysis  and  Results 


Chapter  Overview 

We  created  a  regression  model,  which  would  determine  if  level  five  EYM  data 
was  more  predictive  than  level  three  EYM  data.  We  anticipated  the  ability  to  indicate 
level  five  data,  compared  to  level  three  data,  was  a  better  predictor  for  overall  cost 
growth. 

We  measured  overall  cost  growth  by  calculating  the  growth  of  level  one’s  EAC 
from  the  first  reported  period  to  the  final  period.  Given  our  data  set,  our  regression 
model  includes  service,  program  type,  contractor  and  length  of  contract.  In  order  to 
compare  the  contractor’s  provided  EAC  with  an  EVM  calculated  EAC,  we  divided 
Chapter  Four  into  two  sections.  For  the  purpose  of  the  rest  of  this  chapter,  we  will  refer 
to  the  contractor’s  EAC  as  an  LRE  and  the  recalculated  EAC  as  the  EAC. 

The  first  section  evaluates  the  EAC,  while  the  second  section  will  evaluate  the 
LRE.  Each  of  these  sections  has  the  same  structure.  First  they  will  examine  the 
distribution  of  our  response  variable  (percent  increase  in  level  one  EAC  (or  LRE))  and 
identify  any  outlier  contracts.  Then,  they  both  display  the  multiple  regression  results. 

The  results  also  display  any  contracts  excluded  either  because  it  was  an  extreme  outlier  or 
was  overly  influential  in  the  model. 

Evaluating  the  EAC 

In  the  following  sections,  we  examine  the  results  using  the  EAC. 
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Distribution  of  Response  Variable 


While  conducting  our  analysis,  we  detected  two  extreme  outlier  contracts  based 
upon  level  one  recalculated  EAC  growth.  Contract  N00019-06-G-0001  and  W15P7T-04- 


D-G205  grew  11,606%  and  1,405%  respectively.  Figure  10  displays  the  distribution  of 


the  percent  change  in  level  one’s  EAC. 


Figure  10  -  Distribution  of  Percent  Change  of  Level  1  EAC 


We  removed  these  two  contracts  from  our  data  set.  This  lowered  our  total  number  of 
contracts  for  our  analysis  down  to  thirty-nine.  Figure  1 1  represents  the  distribution 


excluding  the  two  contracts. 
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Figure  11  -  Distribution  of  Percent  Change  of  level  1  EAC  (excluding  two  extreme 


outliers) 


After  removing  the  two  extreme  outliers,  three  additional  contracts  became  outliers. 


Figure  12  displays  our  distribution  removing  these  additional  three  outlier  contracts. 


Figure  12  -  Distribution  of  Percent  Change  of  level  1  EAC  (excluding  three 


additional  outliers) 
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After  removing  the  three  additional  outliers,  one  additional  outlier  was  recognized  and 


removed. 


Figure  13  -  Distribution  of  Percent  Change  of  level  1  EAC  (excluding  all  outliers) 

Figure  13  displays  the  distribution  of  the  response  variable  excluding  all  outliers.  By 
excluding  these  six  contracts,  it  lowered  our  mean  EAC  growth  from  404%  to  46%  and 
lowered  the  standard  deviation  from  1832%  to  59%.  We  identified  six  outliers,  however, 
within  our  models  we  initially  only  excluded  the  two  extreme  outliers.  Table  15  details 
the  six  outlier  contracts  and  the  percent  change  in  level  one  EAC. 
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Table  15  -  EAC  Contract  Outliers 


%  change 
Ivl  1  EAC 

Service 

Project  Name 

Prime  Contract  Number 

11606.53% 

Navy 

H-l  UPGRADES  (4BW/4BN)  -  United  States  Marine 
Corps  Mid-life  Upgrade  to  AH-1W  Attack  Helicopter 
and  UH-1N  Utility  Helicopter 

N  000 19- 06-  G- 000 1 

1405.37% 

Army 

FBCB2  -  Force  XXI  Battle  Command  Brigade  and 
Below  Program 

W 15  P  7T-  04-  D-  G  205 

549.06% 

Navy 

VTUAV  -  Vertical  Takeoff  and  Land  Tactical 
Unmanned  Air  Vehicle  (Fire  Scout) 

N  00019- 12-C-0059 

451.70% 

Navy 

V-22- OSPREY  Joint  Advanced  Vertical  Lift  Aircraft 

N  0042 1- 10-  D-  00 12 

373.09% 

Navy 

V- 22-  OSPREY  Joint  Advanced  Vertical  Lift  Aircraft 

N  61340- ll-C-0004 

225.30% 

Air 

Force 

ISPAN  -  Integrated  Strategic  Planning  and  Analysis 
Network  -  Block  1 

FA8722-04-C-0009 

The  first  two  contracts  in  Table  15  are  the  two  extreme  outliers,  while  the  next  four  were 
the  subsequent  outliers.  Next,  we  briefly  explain  each  outlier. 

The  HI  upgrade  was  the  most  extreme  outlier  in  our  data  set.  Figure  14  displays 
the  contract’s  EAC  along  with  the  BAC.  There  are  two  reasons  for  the  large  percent 
increase  in  level  one  EAC.  The  first  reason  was  because  its  first  EAC  and  BAC  were 
extremely  low.  The  initial  EAC  was  $702,478  and  the  initial  BAC  was  $1,375,089.  Two 
months  later,  both  of  these  figures  increased  to  $27,591,793  and  $19,151,617 
respectively.  Figure  14  also  identifies  a  change  in  scope  in  October  2009,  which  is 
another  cause  for  the  increase  in  cost. 
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HI  Upgrades  (N00019-06-G-0001) 
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80,000,000 

70,000,000 


Figure  14  -  HI  Upgrade 

The  next  contract  excluded  was  the  FBCB2.  Figure  15  displays  both  the  EAC 
and  BAC  for  this  contract.  Unlike  the  HI  upgrade,  a  constant  change  in  quantity  ordered 
caused  this  contract’s  cost  growth.  This  contract’s  cost  has  been  increasing  at  a  constant 
rate  since  it  began. 


FBCB2  (W15P7T-04-D-G205) 
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Figure  15  -  FBCB2 
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Excluding  one  of  the  Navy’s  VTUAV  contracts,  Figure  16  displays  both  the  EAC 
and  BAC.  An  early  change  in  scope  and/or  additional  requirements  explains  the  reason 


for  the  cost  growth  in  this  contract. 


VTUAV  (N00019-12-C-0059) 
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Figure  16  -  VTUAV  (N00019-12-C-0059) 

The  next  two  contracts  excluded  came  from  the  V-22  OSPREY  program.  The  first  V-22 
contract  appears  to  have  had  cost  overruns  since  the  contract  began. 
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V-22  -  Osprey  Joint  Advanced  Vertical  Aircraft 
(N00421-10-D-0012) 
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Figure  17  -  V-22  -  Osprey  Joint  Advanced  Vertical  Aircraft  (N00421-10-D-0012) 

The  second  V-22  contract  shows  a  large  growth  in  EAC  due  to  the  EVM  data  inputted  the 
first  month.  However,  we  did  not  remove  the  first  month’s  data  because  we  wanted  to 
compare  the  EVM  data. 


V-22  -  Osprey  Joint  Advanced  Vertical  Aircraft 
(N  61340-1  l-C-0004) 
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Figure  18  -  V-22  -  Osprey  Joint  Advanced  Vertical  Aircraft  (N61340-1  l-C-0004) 
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The  final  contract  excluded  was  the  ISPAN  contract.  This  contract,  similar  to  the  FBCB2 


contract,  has  had  a  constant  cost  growth  since  the  beginning  of  the  contract. 


ISPAN  (FA8722-04-C-0009) 
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Figure  19  -  ISPAN  (FA8722-04-C-0009) 

Multiple  Regression  Results  using  EAC 

For  each  definition,  we  created  a  twenty  multiple  regression  models,  one  for  every 
5%  complete,  starting  at  5%  and  ending  at  100%.  Figure  20  displays  the  regression 
equation  for  growth  definitions  #1  and  #2.  Figure  21  displays  the  regression  equation  for 
growth  definitions  #3  and  #4.  Figure  22,  illustrates  how  many  contracts  had  cost  data  in 
each  percent  complete  bin. 
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^(y)  —  P 0  +  #1^1  +  $2^2 

Definition  til 

Definition  ti2 

Y=  Level  1  EAC  Change 

Y=  Level  1  LRE  Change 

Xi  =  Level  3  Explanatoiy  Variable 

Xi  —  Level  3  Explanatoiy  Variable 

Xz  —  Level  5  Explanatoiy  Variable 

Xz  —  Level  5  Explanatoiy  Variable 

Figure  20  -  Regression  Equation  for  Growth  Definitions  #1  and  #2 


E(y)  =  Po  +  PlXl  +  p2X2  +  p3X3  +  p4X4  +  ■■■  +  PssX5S 


Definition  #3 

Y=  Level  1 EAC  Change 

Xi  —  Level  3  Explanatoiy  Variable 

Xz  —  Level  5  Explanatory  Variable 
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=  ^-dentVariaUeslistedinA^dixC 

71=3 


Definition  #4 

Y=  Level  1 LRE  Change 

Xi  =  Level  3  Explanatoiy  Variable 

Xz  —  Level  5  Explanatoiy  Variable 
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^  K„  =  Independent  Variables  listed  in  Appendix  C 

71=3 


Figure  21  -  Regression  Equation  for  Growth  Definitions  #3  and  #4 
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Number  of  Contracts  by  Bin 

30 


Figure  22  -  Number  of  Contracts  by  Bin 

For  each  of  our  explanatory  variables,  we  ran  twenty  different  multiple  regression 
models,  one  model  for  each  percent  complete  bin. 

Since  this  is  an  early  attempt  into  examining  EVM  data  below  level  three,  we 
were  not  sure  which  elements  were  significant.  We  used  stepwise  regression  for  all  of 
our  models  in  order  to  determine  which  independent  variables  to  include.  Each  model 
had  the  potential  to  include  our  level  three  and  five  variable  and  fifty  additional 
independent  variables.  Appendix  C  contains  a  list  of  the  additional  independent 
variables.  However,  upon  running  stepwise  regression  for  definitions  #1  and  #2,  the 
stepwise  regression  did  not  include  our  level  three  and  five  explanatory  variables.  After 
we  learned  of  this,  we  only  included  the  level  three  and  five  explanatory  variables  for  our 
model  using  definition  #1  and  #2.  In  the  following  sections,  we  will  reveal  the  results  for 
each  definition  of  cost  growth. 
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Cost  Growth  Definition  #1  Results 


Table  16  details  the  results  of  the  twenty  multiple  regression  models  ran  using 
definition  #1.  The  second  column  represents  the  number  of  iterations  conducted  for  each 
percent  complete.  Multiple  iterations  where  run  if  Cook’s  D  detected  at  least  one 
contract  that  was  overly  influential.  An  overly  influential  contract  is  defined  as  contract 
with  a  Cook’s  D  greater  than  0.5.  We  excluded  overly  influential  contracts  and  repeated 
the  model.  Once  complete  with  the  model,  any  overly  influential  contracts  would  then  be 
included  for  the  next  percent  complete  model. 

The  sixth  column  represents  the  number  of  observations  within  model.  An 
observation  represents  one  contract.  The  final  column  displays  the  number  of 
independent  variables  used  in  the  model.  For  definition  #1,  this  value  will  not  be  larger 
than  two  because  all  the  models  were  limited  to  level  three  and  five  explanatory 
variables.  At  the  95%  percent  complete  bin,  we  used  only  one  independent  variable 
because  the  cumulative  change  for  level  three  and  five  had  a  high  level  of 
multicollinearity.  Appendix  D  has  the  complete  JMP®  output  and  Cook’s  D  for  each  of 
the  models. 
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Table  16  -  Cost  Growth  Definition  #1  JMP®  Results 


Percent  Complete 

Number  of 

Interations 

R2 

Adjusted  R2 

Prob  >  F 

Number  of 

Observations 

#IV 

5% 

1 

0.008234 

-0.10844 

0.9321 

20 

2 

10% 

2 

0.025397 

-0.09643 

0.814 

19 

2 

15% 

2 

0.031998 

-0.07556 

0.7463 

21 

2 

20% 

3 

0.094713 

-0.00587 

0.4084 

21 

2 

25% 

3 

0.018638 

-0.0795 

0.8285 

23 

2 

30% 

3 

0.032951 

-0.05915 

0.7034 

24 

2 

35% 

3 

0.058247 

-0.03144 

0.5325 

24 

2 

2 

0.159495 

0.089453 

0.1243 

27 

2 

1 

0.097554 

0.02235 

0.2918 

27 

2 

50% 

4 

0.00382 

-0.08674 

0.9588 

25 

2 

55% 

1 

0.064663 

-0.01328 

0.4483 

27 

2 

60% 

2 

0.023485 

-0.06952 

0.7792 

24 

2 

65% 

1 

0.018631 

-0.07058 

0.8131 

25 

2 

70% 

2 

0.034704 

-0.06183 

0.7024 

23 

2 

75% 

2 

0.152431 

0.07171 

0.1761 

24 

2 

80% 

2 

0.33358 

-0.0507 

0.6769 

26 

2 

85% 

1 

0.179567 

0.101431 

0.1252 

24 

2 

90% 

1 

0.034002 

-0.08675 

0.7582 

19 

2 

95% 

2 

0.19763 

0.117393 

0.1476 

12 

1 

100% 

4 

1 

1 

3 

2 

Table  17  lists  all  the  contracts  removed  while  examining  definition  #1.  The  fourth 
column  displays  the  total  number  of  times  a  contract  exclusion  occurred  and  the  final 
column  displays  the  excluded  bins  for  each  contract. 
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Table  17  -  Cost  Growth  Definition  #1  Influential  Contracts 


Service 

Program 

Contract  Number 

Total 

Excluded  Bins 

Navy 

AGM-88E  AARGM  -  AGM-88E  Advanced  Anti-Radiation  Guided  Missile 
(AARGM)  Program 

N00019-03-C-0353 

1 

100% 

Navy 

CEC- Cooperative  Engagement  Capability 

N 00024- 05-C-5100 

1 

100% 

Army 

Chem  Demil  -  CMA 

D  ACA87- 89-  C- 007 6 

1 

95% 

Navy 

DDG  1000  -  ZUMWALT  CLASS  Destroyer 

N 00024- 05-  C-  5346 

1 

75% 

Navy 

DDG  51-  ARLEIGH  BURKE  CLASS  Guided  Missile  Destroyer 

N 00024- 02-  C-  2304 

2 

70% 

75% 

Army 

EXCALIBUR  -  Family  of  Precision,  155mm  Projectiles 

DA  A  E30- 98-  C- 1032 

1 

20% 

Army 

FBCB2  -  Force  XXI  Battle  Command  Brigade  and  Below  Program 

W15P7T-04-D-G205 

20 

ALL 

Navy 

H-l  UPGRADES  (4BW/4BN)  -  United  States  Marine  Corps  Mid-life 
Upgrade  to  AH-1W  Attack  Helicopter  and  UH-1N  Utility  Helicopter 

N 00019- 06-  G- 0001 

20 

ALL 

Navy 

LCS  -  Littoral  Combat  Ship 

N 00024- 03-  C-  23 10 

3 

20% 

25% 

100% 

Navy 

LCS  -  Littoral  Combat  Ship 

N 00024- 1 1-  C-  2301 

4 

15% 

20% 

25% 

30% 

Navy 

LPD  17  -  SAN  ANTONIO  CLASS  Amphibious  Transport  Dock  Ship 

N 00024- 04-  C- 2204 

1 

60% 

Navy 

MH-60R-  Multi-Mission  Helicopter  Upgrade 

N00019-04-C-0130 

2 

20% 

50% 

Navy 

V- 22-  OSPREY  Joint  Advanced  Vertical  Lift  Aircraft 

N 00421- 10-  D-0012 

2 

10% 

35% 

Navy 

V- 22  -  OSPREY  Joint  Advanced  Vertical  Lift  Aircraft 

N  6 1340- 1 1-  C- 0004 

4 

30% 

35% 

40% 

50% 

Navy 

VTUAV  -  Vertical  Takeoff  and  Land  Tactical  Unmanned  Air  Vehicle 
(Fire  Scout) 

N 000 19- 00-  C- 0277 

3 

80% 

95% 

100% 

Navy 

VTUAV  -  Vertical  Takeoff  and  Land  Tactical  Unmanned  Air  Vehicle 
(Fire  Scout) 

N 000 19- 12-  C- 0059 

7 

10%  -  35% 

50% 

Air  Force 

WGS- Wideband  Global  SATCOM  Program 

F  A8808- 06-  C- 0001 

2 

100% 

Due  to  the  high  p-value  for  the  F  test,  definition  #1  provided  no  useful  models  for 
our  research. 

Cost  Growth  Definition  #3  Results 


Method  to  Select  Independent  Variables 

For  definition  #3,  we  ran  a  stepwise  regression  for  each  model  to  determine  which 
variables  were  significant  at  each  percent  complete  bin.  The  only  exception  was  at  70% 
complete.  Stepwise  regression  was  not  able  to  create  a  model  at  70%.  Therefore,  we 
manually  created  the  model  by  using  our  level  three  explanatory  variable  and  the  square 
of  our  level  three  explanatory  variable.  Figure  23  illustrates  the  number  of  variables  each 
model  used,  as  well  as  the  number  of  variables  found  significant  in  each  percentage 
complete  bin. 
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Definition  #3  Independent  Variables 


20 


H#  Indpendent  Variables 
Q  #  Significant  IV 


Figure  23  -  Cost  Growth  Definition  #3  Independent  Variables  by  Percent  Complete 

Table  18  displays  the  results  of  the  twenty  multiple  regression  models  ran  using 
definition  #3.  The  first  seven  columns  are  structured  the  same  as  the  first  seven  columns 
of  Table  16.  The  three  additional  columns  contain  the  number  of  significant  variables 
found  in  each  model  and  if  the  model  contained  either  of  our  level  three  or  level  five 
explanatory  variable.  Appendix  F  has  the  complete  JMP®  output  for  each  of  the  models. 
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Table  18  -  Cost  Growth  Definition  #3  JMP®  Results 


Percent  Complete 

Number  of 

Interations 

rcc 

Adjusted  R2 

Prob  >  F 

Number  of 

Observations 

Al# 

#  Significant  IV 

Include  Level  3 

variable  (Y/N) 

Include  Level  5 

variable  (Y/N) 

5% 

4 

0.500839 

0.084872 

0.3498 

34 

15 

0 

N 

N 

10% 

3 

0.225539 

0.178602 

0.0147 

36 

2 

1 

N 

N 

15% 

3 

0.167795 

0.144018 

0.0118 

37 

1 

1 

N 

N 

20% 

6 

0.747205 

0.397181 

0.0841 

32 

18 

0 

N 

N 

25% 

3 

0.178451 

0.15563 

0.0082 

38 

1 

1 

N 

N 

30% 

7 

0.515625 

0.463728 

0.0001 

32 

3 

2 

N 

N 

35% 

3 

0.308683 

0.248569 

0.0143 

26 

2 

1 

Y 

N 

40% 

3 

0.178451 

0.15563 

0.0082 

38 

1 

1 

N 

N 

45% 

9 

0.993052 

0.988421 

0.0001 

11 

4 

3 

Y 

N 

50% 

10 

0.991076 

0.982151 

0.0002 

9 

4 

3 

Y 

N 

55% 

7 

0.996923 

0.993406 

0.0001 

16 

8 

6 

Y 

N 

60% 

6 

0.906008 

0.843347 

0.0004 

16 

6 

4 

N 

Y 

65% 

2 

0.993305 

0.984379 

0.0001 

22 

12 

9 

Y 

N 

70% 

** 

0.752003 

0.725898 

0.0001 

22 

2 

1 

Y 

N 

75% 

3 

0.994692 

0.988853 

0.0001 

22 

11 

9 

Y 

N 

80% 

3 

0.968438 

0.950402 

0.0001 

23 

8 

4 

Y 

N 

85% 

2 

0.257657 

0.147681 

0.0802 

32 

4 

1 

N 

N 

90% 

2 

0.046627 

-0.08049 

0.8303 

35 

4 

0 

N 

N 

95% 

2 

0.145953 

-0.0602 

0.6654 

37 

7 

0 

N 

N 

100% 

1 

0.034283 

-0.11661 

0.948 

38 

5 

0 

N 

N 

Table  19  lists  all  the  contracts  removed  while  examining  definition  #3.  The  fourth 
column  displays  the  total  number  of  times  a  contract  exclusion  occurred  and  the  final 
column  displays  the  excluded  bins  for  each  contract. 
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Table  19  -  Cost  Growth  Definition  #3  Influential  Contracts 


Service 

Program 

Contract  Number 

Total 

Excluded  Bins  | 

Navy 

AAG  -  Advanced  Arresting  Gear  Program 

N  683 35- 03-  C- 0205 

4 

45% 

50% 

60% 

65% 

Navy 

AMDR  -  Air  &  Missile  Defense  Radar 

N00024-10-C-5359 

1 

70% 

Air  Force 

B-2  MOP  -  Massive  Ordnance  Penetrator 

F33657-99-D-0028 

5 

45%  -  60% 

70% 

Army 

Chem  Demil  -  CMA 

D  ACA87- 89-  C- 007 6 

2 

50% 

70% 

Army 

Chem  Demil  -  CMA 

D  A  A  A09- 97-  C-0025 

3 

55% 

70% 

85% 

Navy 

DDG  1000  -  ZUMWALT  CLASS  Destroyer 

N 00024- 05-  C-  5346 

3 

45% 

55% 

70% 

Navy 

DDG  51-  ARLEIGH  BURKE  CLASS  Guided  Missile  Destroyer 

N 00024- 02-  C-  2304 

2 

50% 

70% 

Navy 

EA-18G  -  Airborne  Electronic  Attack  variant  of  the  F/A- 18  aircraft 

N 000 19- 04-  C- 0005 

2 

45% 

70% 

Army 

EXCALIBUR  -  Family  of  Precision,  155mm  Projectiles 

DA  A  E30- 98-  C- 1032 

2 

50% 

55% 

Army 

EXCALIBUR  -  Family  of  Precision,  155mm  Projectiles 

W15QKN-08-C-0530 

2 

45% 

70% 

Army 

FBCB2  -  Force  XXI  Battle  Command  Brigade  and  Below  Program 

W15P7T-04-D-G205 

20 

All 

Navy 

H-l  UPGRADES  (4BW/4BN)  -  United  States  Marine  Corps  Mid-life 
Upgrade  to  AH-1W  Attack  Helicopter  and  UH-1N  Utility  Helicopter 

N 00019- 06-  G- 0001 

20 

All 

Army 

IAMD  -  Integrated  Air  &  Missile  Defense 

W31P4Q-08-C-0418 

2 

50% 

70% 

Air  Force 

ISPAN  -  Integrated  Strategic  Planning  and  Analysis  Network  -  Block  1 

F  A8722- 04-  C- 0009 

3 

30% 

50% 

70% 

Army 

JAGM -Joint  Air-to-Ground  Missile 

W31P4Q-08-C-A123 

1 

70% 

Navy 

JPALS  -  Joint  Precision  Approach  and  Landing  System 

N 000 19- 08-  C- 0034 

1 

55% 

Navy 

JSOW  (BASELINE/UNITARY)  -  Joint  Stand-Off  Weapon  Baseline 
Variant  and  Unitary  Warhead  Variant 

N 000 19-05-  G- 0008 

2 

45% 

50% 

Army 

JTRS  GMR-JointTactical  Radio  System  Ground  Mobile  Radio 

DA  A  B07- 02-  C-  C403 

10 

5%  -  20% 

50%  -  60% 

70% 

85%  -  90% 

Navy 

LCS  -  Littoral  Combat  Ship 

N 00024- 03-C-2310 

12 

20% 

30% 

45%  -  60% 

70%  -  95% 

Navy 

LPD  17-  SAN  ANTONIO  CLASS  Amphibious  Transport  Dock  Ship 

N 00024- 04-  C- 2204 

3 

55% 

65% 

70% 

Navy 

MH-60R-  Multi-Mission  Helicopter  Upgrade 

N00019-04-C-0130 

3 

45% 

70% 

85% 

Navy 

MH-60R  -  Multi-Mission  Helicopter  Upgrade 

N 000 19- 08-  C- 0005 

5 

45%  -  50% 

70%  -  75% 

85% 

Navy 

MH-60R  -  Multi-Mission  Helicopter  Upgrade 

N 000 19- 09-  C- 0059 

2 

10% 

65% 

Navy 

MH-60S  -  Multi-Mission  Combat  Support  Helicopter 

N 000 19- 03-  G- 0003 

4 

30% 

55% 

70% 

80% 

Air  Force 

MPS  -  Mission  Planning  System 

FA8720-04-  D-0005 

2 

55% 

70% 

Navy 

MQ-4C  Triton  (Formerly  BAMS) 

N 000 19- 08-  C- 0023 

2 

45% 

50% 

Navy 

P-8A-  Poseidon  Program 

N 000 19- 04-  C-  3 146 

1 

70% 

Navy 

SM-6-Standard  Missile-6 

N 00024- 09-  C-  5305 

2 

45% 

50% 

Navy 

V- 22-  OSPREY  Joint  Advanced  Vertical  Lift  Aircraft 

N 00421- 10-  D-0012 

9 

20% 

30% 

45%  -  60% 

70%  -  80% 

Navy 

V- 22-  OSPREY  Joint  Advanced  Vertical  Lift  Aircraft 

N  6 1340- 1 1-  C- 0004 

7 

5% 

20% 

30% 

45%  -  60% 

Navy 

V-22  -  OSPREY  Joint  Advanced  Vertical  Lift  Aircraft 

N  61339-08- D-0004 

5 

5% 

20% 

55% 

70%  80% 

Navy 

VTUAV  -  Vertical  Takeoff  and  Land  Tactical  Unmanned  Air  Vehicle 
(Fire  Scout) 

N 000 19- 00-  C- 0277 

6 

45% 

50% 

60% 

70%  -  80% 

Navy 

VTUAV  -  Vertical  Takeoff  and  Land  Tactical  Unmanned  Air  Vehicle 
(Fire  Scout) 

N 000 19- 12-  C- 0059 

7 

5% 

20% 

30% 

45%  -  60% 

Air  Force 

WGS -Wideband  Global  SATCOM  Program 

F  A8808- 06-  C- 0001 

5 

50% 

60%  -  70% 

90% 

Using  definition  #3,  fourteen  of  our  twenty  models  had  an  F-test  less  than  our 
alpha.  In  addition,  nine  of  the  twenty  models  contained  either  our  level  three  or  five 
explanatory  variable.  Out  of  the  nine  models  that  contained  either  our  level  three  or  five 
explanatory  variable,  the  level  three  variable  was  used  in  eight  different  models:  35%, 
45%,  50%,  55%,  65%,  70%,  75%  and  80%,  while  our  level  five  explanatory  variable  was 
used  once  at  60%. 
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Evaluating  the  LRE 

In  the  following  sections,  we  examine  the  results  using  the  LRE. 

Distribution  of  Response  Variable 

While  conducting  our  analysis,  we  detected  two  extreme  outlier  contracts  based 
on  level  one  LRE  growth.  Contract  N00024-09-C-5305  and  N00019-06-G-0001  grew 


13,374%  and  4,346%  respectively.  Figure  24  displays  the  distribution  of  the  percent 


change  in  level  one’s  LRE. 


Figure  24  -  Distribution  of  Percent  Change  of  Level  1  LRE 


After  removing  the  two  extreme  outliers,  three  additional  contracts  became  outliers. 
Figure  25  displays  the  distribution  after  we  removed  the  two  extreme  outliers.  As  Figure 
25  indicates,  there  are  now  three  additional  outliers. 
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Figure  25  -  Distribution  of  Percent  Change  of  Level  1  LRE  (Excluding  two  extreme 


outliers) 


Figure  26  displays  the  distribution  of  the  percent  change  of  level  one  LRE  after  removing 


all  outliers. 


Figure  26  -  Distribution  of  Percent  Change  of  Level  1  LRE  (Excluding  three 


additional  outliers) 
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By  excluding  these  five  contracts,  it  lowered  our  mean  LRE  growth  from  542%  to  54% 
and  lowered  the  standard  deviation  from  21 97%  to  73%.  We  identified  five  outliers, 
however,  in  our  models,  we  initially  only  excluded  the  two  extreme  outliers.  Table  20 
details  the  five  outlier  contracts  and  the  percent  change  in  level  one  EAC. 

Table  20  -  LRE  Contract  Outliers 


%  change 
Ivl  1  EAC 

Service 

Project  Name 

Prime  Contract  Number 

13374.35% 

Navy 

SM-6-Standard  Missile-6 

N00024-09-C-5305 

4346.86% 

Navy 

H-l  UPGRADES  (4BW/4BN)  -  United  States  Marine 
Corps  Mid-life  Upgrade  to  AH-1W  Attack  Helicopter 
and  UH-1N  Utility  Helicopter 

N00019-06-G-0001 

1268.43% 

Army 

FBCB2  -  Force  XXI  Battle  Command  Brigade  and 
Below  Program 

W 15  P  7T-  04-  D-  G  205 

457.44% 

Navy 

V- 22-  OSPREY  Joint  Advanced  Vertical  Lift  Aircraft 

N  0042 1- 10-  D-  00 12 

325.25% 

Navy 

VTUAV  -  Vertical  Takeoff  and  Land  Tactical 
Unmanned  Air  Vehicle  (Fire  Scout) 

N  00019- 12-C-0059 

The  first  two  contracts  in  Table  20  are  the  two  extreme  outliers,  while  the  next  three  were 
the  subsequent  outliers.  Next,  we  briefly  explain  and  compare  each  outlier  with  the 
outliers  using  the  EAC. 

The  most  extreme  outlier  using  the  LRE  was  the  Navy’s  Standard  Missile  -  6 
(N00024-09-C-5305).  This  contract  did  not  register  as  an  outlier  using  the  EAC.  Its 
level  one  EAC  percent  change  was  1 16.04%.  Lack  of  EYM  data  is  the  cause  of  the 
discrepancy.  The  contractor  reported  an  LRE  of  $2,079,916  in  July  2010;  however,  they 
did  not  report  any  EYM  data  until  September  2010  for  $120,489,469.  The  difference  in 
the  beginning  value  can  explain  the  drastic  difference  between  the  percent  growth  of  the 
LRE  and  EAC. 
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The  next  extreme  outlier  was  the  Navy’s  H-l  Upgrade  (N00019-06-G-0001). 


This  contract  grew  1 1,606.53%  using  the  EAC.  Figure  27  displays  the  level  one  LRE 
growth  compared  to  the  level  one  BAC  growth.  The  key  differences  between  the  LRE 
and  EAC  for  this  contract  are  the  starting  and  ending  value.  The  beginning  EAC  and 
LRE  are  $702,478  and  $1,375,089,  respectively.  The  ending  values  are  $82,235,822  and 
$61,148,373,  respectively.  In  addition,  throughout  the  contract,  the  EAC  was 
consistently  higher. 

HI  Upgrade  (N00019-06-G-0001) 

70,000,000 
60,000,000 
50,000,000 
40,000,000 
30,000,000 
20,000,000 
10,000,000 
0 


Figure  27  -  HI  Upgrade  LRE  growth 

The  next  outlier  contract  was  the  Army’s  FBCB2  (W15P7T-04-D-G205).  The 
difference  in  growth  between  the  LRE  and  EAC  is  1405.37%  and  1268.43%.  Using 
either  the  LRE  or  EAC,  the  contract  had  a  constant  growth  since  the  contract  began. 

The  Navy’s  V-22  (N00421-10-D-0012)  was  the  next  outlier.  There  was  virtually 
no  difference  with  the  percent  change  of  the  LRE  and  the  EAC.  The  EAC  grew  451.70% 
and  the  LRE  grew  457.44%. 
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The  last  outlier  was  the  Navy’s  VTUAV  (N00019-12-C-0059).  This  contract  also 
was  an  outlier  using  the  percent  change  in  EAC.  Its  EAC  growth  was  549.06%  compared 
to  325.25%  using  the  percent  change  of  the  LRE.  Figure  28  displays  the  level  one  LRE 


and  BAC  for  this  contract.  As  Figure  28  displays,  there  was  a  change  in  scope  early 
within  the  contract’s  life. 


VTUAV  (N00019-12-C-0059) 
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Figure  28  -  VTUAV  LRE  Growth 

Multiple  Regression  Results  using  LRE 

As  mentioned  earlier  in  this  chapter,  we  used  only  the  level  three  and  five 
explanatory  variables  for  definition  #2  and  we  used  stepwise  regression  to  determine 
which  variables  to  add  for  definition  #4. 

Cost  Growth  Definition  #2  Results 

Table  21  details  the  results  of  the  twenty  multiple  regression  models  ran  applying 
definition  #2.  Similar  to  definition  #1,  we  only  included  the  level  three  and  five  variables 
in  each  model.  Appendix  E  has  the  complete  JMP®  output  for  each  of  the  models. 
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Table  21  -  Cost  Growth  Definition  #2  JMP®  Results 


Percent  Complete 

Number  of 

Interations 

R2 

Adjusted  R2 

Prob  >  F 

Number  of 

Observations 

#IV 

5% 

2 

0.016459 

-0.10648 

0.8757 

19 

2 

7 

0.271891 

0.211215 

0.0558 

14 

1 

4 

0.072469 

0.01449 

0.28 

18 

1 

20% 

1 

0.030431 

-0.05388 

0.7009 

26 

2 

25% 

1 

0.005493 

-0.03594 

0.719 

26 

1 

30% 

7 

0.105754 

0.053151 

0.1743 

19 

1 

35% 

4 

0.029098 

-0.01945 

0.4479 

22 

1 

40% 

3 

0.015432 

-0.07407 

0.8428 

25 

2 

45% 

1 

0.014753 

-0.02466 

0.5462 

27 

1 

50% 

2 

0.049447 

0.008119 

0.2854 

25 

1 

55% 

1 

0.043954 

-0.03572 

0.5831 

27 

2 

60% 

1 

0.011253 

-0.08291 

0.888 

24 

2 

65% 

1 

0.009038 

-0.03405 

0.6512 

25 

1 

70% 

1 

0.003963 

-0.0909 

0.9592 

24 

2 

75% 

1 

0.080927 

0.040967 

0.1681 

25 

1 

80% 

4 

0.002549 

-0.04279 

0.8148 

24 

1 

85% 

1 

0.001908 

-0.04149 

0.8357 

25 

1 

90% 

3 

0.256683 

0.142327 

0.1424 

16 

2 

95% 

1 

0.001877 

-0.0813 

0.8831 

14 

1 

100% 

5 

Excluded  all  contracts 

0 

0 

Table  22  lists  all  the  contracts  removed  while  examining  definition  #2.  The  fourth 
column  displays  the  total  number  of  times  a  contract  exclusion  occurred  and  the  final 
column  displays  the  excluded  bins  for  each  contract.  Due  to  the  high  p- value  for  the  F 
test,  definition  #2  provided  no  useful  models  for  our  research. 
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Table  22  -  Cost  Growth  Definition  #2  Influential  Contracts 


Service 

Program 

Contract  Number 

Total 

Excluded  Bins 

Navy 

AGM-88E  AARGM  -  AGM-88E  Advanced  Anti-Radiation  Guided  Missile 
(AARGM)  Program 

N00019-03-C-0353 

1 

100% 

Navy 

AMDR  -  Air  &  Missile  Defense  Radar 

N 00024- 10- C- 5359 

1 

80% 

Navy 

CEC- Cooperative  Engagement  Capability 

N 00024- 05-C-5100 

1 

100% 

Army 

Chem  Demil  -  CMA 

D  ACA87- 89-  C- 007 6 

1 

30% 

Army 

EXCALIBUR  -  Family  of  Precision,  155mm  Projectiles 

W15QKN-08-C-0530 

1 

30% 

Army 

FBCB2  -  Force  XXI  Battle  Command  Brigade  and  Below  Program 

W15P7T-04-D-G205 

7 

10%  - 15% 

30%  -  35% 

50% 

80%  90% 

Navy 

H-l  UPGRADES  (4BW/4BN)  -  United  States  Marine  Corps  Mid-life 
Upgrade  to  AH- 1W  Attack  Helicopter  and  UH-1N  Utility  Helicopter 

N 00019- 06-  G- 0001 

20 

ALL 

Army 

JAGM -Joint  Air-to-Ground  Missile 

W31P4Q-08-C-A123 

2 

15% 

90% 

Navy 

JPALS  -  Joint  Precision  Approach  and  Landing  System 

N 000 19- 08-  C- 0034 

2 

10% 

30% 

Navy 

JSOW  (BASELINE/UNITARY) -Joint  Stand-Off  Weapon  Baseline 
Variant  and  Unitary  Warhead  Variant 

N 000 19-05-  G- 0008 

2 

30% 

35% 

Navy 

LCS  -  Littoral  Combat  Ship 

N 00024- 03-C-2310 

1 

100% 

Navy 

LCS  -  Littoral  Combat  Ship 

N 00024- 1 1-  C-  2301 

2 

15% 

40% 

Navy 

LPD  17-  SAN  ANTONIO  CLASS  Amphibious  Transport  Dock  Ship 

N 00024- 04-  C- 2204 

20 

ALL 

Navy 

MH-60R  -  Multi-Mission  Helicopter  Upgrade 

N00019-04-C-0130 

1 

50% 

Navy 

SM-6-Standard  Missile-6 

N 00024- 09-  C-  5305 

20 

ALL 

Navy 

SM-6-Standard  Missile-6 

N 00024- 04-  C-  5344 

1 

100% 

Navy 

V- 22-  OSPREY  Joint  Advanced  Vertical  Lift  Aircraft 

N 00421- 10-  D-0012 

5 

10% 

30%  -  35% 

40% 

80% 

Navy 

V- 22-  OSPREY  Joint  Advanced  Vertical  Lift  Aircraft 

N  61339-08- D-0004 

1 

100% 

Navy 

VTUAV  -  Vertical  Takeoff  and  Land  Tactical  Unmanned  Air  Vehicle 
(Fire  Scout) 

N 000 19- 00-  C- 0277 

1 

100% 

Air  Force 

WGS- Wideband  Global  SATCOM  Program 

F  A8808- 06-  C- 0001 

1 

100% 

Army 

WIN-T  Inc.  3  -  Warfighter  Information  Network  Tactical  Increment  3 

DA  A  B07- 02-  C-  F404 

1 

90% 

Cost  Growth  Definition  #4  Results 


Method  to  Select  Independent  Variables 

For  definition  #4,  we  ran  a  stepwise  regression  for  each  model  to  determine  which 
variables  were  significant  at  each  percent  complete  bin.  The  only  exception  was  at  90% 
complete.  Stepwise  regression  was  not  able  to  create  a  model  at  90%.  Therefore,  we 
manually  created  the  model  by  using  our  level  three  explanatory  variable  and  the  square 
of  our  level  three  explanatory  variable.  Table  23  illustrates  the  results  of  the  twenty 
multiple  regression  models  ran  applying  definition  #4.  Appendix  H  has  the  complete 
JMP®  output  for  each  of  the  models. 
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Table  23  -  Cost  Growth  Definition  #4  JMP®  Results 


Percent  Complete 

Number  of 

Interations 

rcc 

Adjusted  R2 

Prob  >  F 

Number  of 

Observations 

Al# 

#  Significant  IV 

Include  Level  3 

variable  (Y/N) 

Include  Level  5 

variable  (Y/N) 

5% 

1 

0.147679 

0.096023 

0.0716 

36 

2 

0 

N 

N 

10% 

1 

0.18978 

0.165954 

0.0079 

36 

1 

1 

N 

N 

15% 

1 

0.251717 

0.206366 

0.0084 

36 

2 

1 

N 

N 

20% 

5 

0.644684 

0.348587 

0.1014 

23 

10 

0 

N 

N 

25% 

3 

0.55403 

0.451114 

0.001 

33 

6 

1 

N 

N 

30% 

1 

0.071065 

0.014766 

0.2963 

36 

2 

0 

N 

N 

35% 

2 

0.949797 

0.698785 

0.1026 

25 

20 

0 

N 

N 

40% 

6 

0.991617 

0.984166 

0.0001 

18 

8 

6 

N 

Y 

45% 

2 

0.60629 

0.570499 

0.0001 

25 

2 

2 

Y 

N 

50% 

7 

0.955606 

0.947535 

0.0001 

14 

2 

2 

N 

Y 

55% 

4 

0.999476 

0.998951 

0.0001 

15 

7 

6 

Y 

N 

60% 

1 

0.992805 

0.983211 

0.0001 

22 

12 

9 

Y 

N 

65% 

3 

0.880398 

0.861998 

0.0001 

16 

2 

1 

Y 

N 

70% 

9 

0.038553 

-0.03012 

0.4661 

16 

1 

0 

N 

N 

75% 

2 

0.994196 

0.987102 

0.0001 

21 

11 

7 

Y 

N 

80% 

1 

0.924452 

0.893344 

0.0001 

25 

7 

4 

Y 

N 

85% 

4 

0.901075 

0.89448 

0.0001 

17 

1 

1 

Y 

N 

90% 

** 

0.919212 

0.907671 

0.0001 

17 

2 

1 

Y 

N 

95% 

2 

0.857263 

0.643157 

0.1002 

11 

6 

0 

N 

N 

100% 

2 

0.099865 

0.006748 

0.3761 

33 

3 

0 

N 

N 

Table  24  lists  all  the  contracts  removed  while  examining  definition  #4.  The  fourth 
column  displays  the  total  number  of  times  a  contract  exclusion  occurred  and  the  final 
column  displays  the  excluded  bins  for  each  contract. 
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Table  24  -  Cost  Growth  Definition  #4  Influential  Contracts 


Service 

Program 

Contract  Number 

Total 

Excluded  Bins 

Navy 

AAG  -  Advanced  Arresting  Gear  Program 

N  68335-03-  C-0205 

5 

50% 

65% 

70% 

85%  -  90% 

Navy 

AGM-88E  AARGM  -  AGM-88E  Advanced  Anti-Radiation  Guided  Missile 
(AARGM)  Program 

N00019-03-C-0353 

2 

85% 

100% 

Navy 

AMDR  -  Air  &  Missile  Defense  Radar 

N 00024- 10- C- 5359 

1 

90% 

Air  Force 

B-2  MOP  -  Massive  Ordnance  Penetrator 

F33657-99-D-0028 

3 

50% 

65% 

85% 

Navy 

CEC- Cooperative  Engagement  Capability 

N 00024- 05-  C-  5 100 

2 

90% 

95% 

Army 

Chem  Demil  -  CMA 

DACA87-89-C-0076 

5 

20% 

40% 

65%  -  70% 

90% 

Army 

Chem  Demil  -  CMA 

DAAA09-97-C-0025 

4 

50% 

55% 

70% 

90% 

Navy 

DDG  1000-  ZUMWALT  CLASS  Destroyer 

N 00024-05-  C-  5346 

4 

40% 

70% 

75% 

90% 

Navy 

DDG  51-  ARLEIGH  BURKE  CLASS  Guided  Missile  Destroyer 

N 00024-02-  C-  2304 

1 

70% 

Navy 

EA-18G  -  Airborne  Electronic  Attack  variant  of  the  F/A- 18  aircraft 

N 000 19- 04-  C- 0005 

4 

20% 

50% 

55% 

90% 

Army 

EXCALIBUR  -  Family  of  Precision,  155mm  Projectiles 

DAAE30-98-C-1032 

2 

70% 

90% 

Army 

EXCALIBUR  -  Family  of  Precision,  155mm  Projectiles 

W15QKN-08-C-0530 

1 

70% 

Army 

FBCB2-  Force  XXI  Battle  Command  Brigade  and  Below  Program 

W15P7T-04-D-G205 

20 

ALL 

Navy 

H-l  UPGRADES  (4BW/4BN)  -  United  States  Marine  Corps  Mid-life 
Upgrade  to  AH-1W  Attack  Helicopter  and  UH-1N  Utility  Helicopter 

N00019-06-G-0001 

20 

ALL 

Army 

IAMD  -  Integrated  Air  &  Missile  Defense 

W31P4Q-08-C-0418 

2 

20% 

70% 

Air  Force 

ISPAN  -  Integrated  Strategic  Planning  and  Analysis  Network  -  Block  1 

F  A8722- 04-  C- 0009 

6 

20% 

50%  -  70% 

85% 

Army 

JAGM -Joint  Air-to-Ground  Missile 

W31P4Q-08-C-A123 

2 

65% 

90% 

Navy 

JPALS- Joint  Precision  Approach  and  Landing  System 

N 000 19- 08-  C- 0034 

1 

40% 

Navy 

JPALS- Joint  Precision  Approach  and  Landing  System 

N 000 19- 08-  C- 0034 

2 

20% 

25% 

Navy 

JSOW  (BASELINE/UNITARY)  -  Joint  Stand-Off  Weapon  Baseline 
Variant  and  Unitary  Warhead  Variant 

N 000 19- 05-  G- 0008 

4 

20% 

55% 

70% 

90% 

Army 

JTRS  GMR -Joint  Tactical  Radio  System  Ground  Mobile  Radio 

DA  A  B07- 02-  C-  C403 

4 

50% 

55% 

70% 

90% 

Navy 

LCS  -  Littoral  Combat  Ship 

N 00024-03- C- 23 10 

8 

20%  40% 

50%  -  55% 

70% 

85%  -  95% 

Navy 

LCS  -  Littoral  Combat  Ship 

N 00024- 1 1-  C-  2301 

2 

40% 

70% 

Navy 

LPD  17  -  SAN  ANTONIO  CLASS  Amphibious  Transport  Dock  Ship 

N 00024-04- C- 2204 

20 

ALL 

Navy 

MH-60R  -  Multi-Mission  Helicopter  Upgrade 

N00019-04-C-0130 

3 

40% 

50% 

70% 

Navy 

MH-60R  -  Multi-Mission  Helicopter  Upgrade 

N 000 19- 08-  C- 0005 

4 

20% 

40% 

50% 

70% 

Navy 

MH-60R  -  Multi-Mission  Helicopter  Upgrade 

N 000 19- 09-  C- 0059 

2 

20% 

55% 

Navy 

MH-60S  -  Multi-Mission  Combat  Support  Helicopter 

N 000 19- 03-  G- 0003 

1 

50% 

Navy 

MQ-4C Triton  (Formerly  BAMS) 

N 000 19- 08-  C- 0023 

1 

70% 

Navy 

SM-6-Standard  Missile-6 

N 00024-09- C- 5305 

20 

ALL 

Navy 

SM-6-Standard  Missile-6 

N 00024-04- C- 5344 

1 

90% 

Navy 

V-22  -  OSPREY  Joint  Advanced  Vertical  Lift  Aircraft 

N 0042 1-10- D- 00 12 

16 

20% 

25% 

35%  - 100% 

Navy 

V-22  -  OSPREY  Joint  Advanced  Vertical  Lift  Aircraft 

N  6 1340- 1 1-  C- 0004 

2 

20% 

55% 

Navy 

V-22  -  OSPREY  Joint  Advanced  Vertical  Lift  Aircraft 

N  61339-08- D-0004 

2 

65% 

75% 

Navy 

VTUAV  -  Vertical  Takeoff  and  Land  Tactical  Unmanned  Air  Vehicle 
(Fire  Scout) 

N 000 19- 00-  C- 0277 

2 

70% 

90% 

Navy 

VTUAV  -  Vertical  Takeoff  and  Land  Tactical  Unmanned  Air  Vehicle 
(Fire  Scout) 

N 000 19- 12-  C- 0059 

5 

20% 

25% 

50%  -  55% 

70% 

Air  Force 

WGS- Wideband  Global  SATCOM  Program 

F  A 8808- 06-  C- 0001 

8 

20% 

50%  -  70% 

85%  -  90% 

100% 

Army 

WIN-T  Inc.  3  -  Warfighter  Information  Network  Tactical  Increment  3 

DAAB07-02-C-F404 

1 

90% 

Using  definition  #4,  thirteen  of  our  twenty  models  had  an  F-test  less  than  our 
alpha.  In  addition,  ten  of  the  twenty  models  contained  either  our  level  three  or  level  five 
explanatory  variable.  Out  of  the  ten  models  that  contained  either  our  level  three  or  level 
five  explanatory  variable,  the  level  three  variable  was  used  in  eight  different  models: 
45%,  55%,  60%,  65%,  75%,  80%,  85%  and  90%,  while  our  level  five  explanatory 
variable  was  used  twice  at  40%  and  50%. 
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Comparing  Level  Three  EVM  Data  with  Level  Five  EVM  Data 


While  conducting  our  analysis  of  lower  level  EVM  data,  we  noticed  a  high 
correlation  between  the  contractor’s  level  three  and  five  data.  In  order  to  verify  this,  we 
conducted  a  Fit  Y  by  X  comparing  the  level  three  variable  to  the  level  five  variable.  We 
conducted  this  test  using  both  the  contractor’s  EAC  and  the  recalculated  EAC.  Figure  29 
displays  the  results  comparing  the  levels  three  and  five  using  the  LRE.  The  slope  of  the 
fit  line  for  this  bin  was  .9983099,  indicating  almost  perfect  correlation  between  the  level 
three  and  level  five  variables. 


H*  Thesis  JMP  File  -  Explanatory  Variable  #4  -  Fit  Y  by  X  of  Lvl  3  C... 


n 


x 


®  Bivariate  Fit  of  Lvl  3  Cum  Change  /  Final  Lvl  1  EAC 
45%  By  Lvl  5  Cum  Change  /  Final  Lvl  1  EAC 

45% 

T - Linear  Fit 

A 

Linear  Fit 

Lvl  3  Cum  Change /  Final  Lvl  1  EAC  45%  =  0.0473537  + 

0.99£3090*Lvl  5  C  u  m  C  ha  n  g  e  /  Fi  n  a  I  Lvl  1  EAC  45% 


Figure  29  -  Comparing  Level  three  and  Level  five  LRE  at  45  %  complete 
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Figure  30  displays  the  results  comparing  the  levels  three  and  five  using  the  EAC.  The 
slope  of  the  fit  line  for  this  bin  was  .6470539.  In  contrast  to  the  LRE’s  slope  at  45%, 


very  little  correlation  is  apparent  between  the  level  three  and  level  five  variable. 


I!*  Thesis  JMP  File  -  Explanatory  Variable  #3  -  Fit  Y  by  X  of  Lvl  3  C . 


□ 


x 


A 


®  Bivariate  Fit  of  Lvl  3  Cum  Change  /  Final  Lvl  1  EAC 
45%  By  Lvl  5  Cum  Change  /  Final  Lvl  1  EAC  45% 


Lvl  3  Cum  Change  /  Final  Lvl  1  EAC  45%  =  0.0536095  + 

0,6470 539* Lvl  5  Cum  Change  /  Final  Lvl  1  EAC  45% 


Figure  30  -  Comparing  Level  three  and  Level  five  EAC  at  45%  complete 

Appendix  I  contains  the  complete  results  using  the  contractor’s  EAC  and  Appendix  J 
contains  the  results  using  the  recalculated  EAC. 

We  looked  at  the  slope  of  the  fit  line  to  evaluate  the  relationship  between  level 
three  and  five.  Level  three  and  five  variables  were  identical  the  closer  the  slope  of  the 
line  was  to  one.  Table  25  compares  the  slope  by  percent  complete  for  the  contractor’s 
EAC  and  the  recalculated  EAC.  Figure  31  and  Figure  32  visually  display  the  slopes.  As 
either  the  table  or  figures  indicate,  the  contractor’s  level  three  and  five  variables  were 
almost  identical.  In  addition,  Figure  32,  illustrates  a  large  increase  of  the  slope  in  the 
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100%  complete  bin.  Contract  FA8808-06-C-0001  caused  this  sudden  increase  at  100%. 


If  this  contract  is  excluded  the  slope  would  decrease  to  0.3033184. 


Table  25  -  Comparing  Slopes  between  Level  Three  and  Five  with  Respects  to  the 

response  of  EAC  and  LRE 


LRE 

EAC 

5% 

1.028042 

0.020022 

10% 

0.971286 

0.365103 

15% 

1.111285 

0.662947 

20% 

1.113232 

0.085429 

25% 

1.088484 

0.555081 

30% 

1.071052 

0.482034 

35% 

1.126062 

1.052112 

40% 

1.117613 

0.651423 

45% 

0.99831 

0.647054 

50% 

0.976149 

0.610131 

55% 

0.901014 

0.565287 

60% 

0.886609 

0.57314 

65% 

0.922645 

0.561618 

70% 

0.937097 

0.567915 

75% 

0.969642 

0.622586 

80% 

0.960558 

0.528834 

85% 

0.931023 

0.577463 

90% 

0.933059 

0.501941 

95% 

0.891078 

0.554161 

100% 

1.205188 

1.549535 

Mean 

1.006971 

0.586691 

Std  Dev 

0.093836 

0.308297 
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Figure  31  -  LRE  Slope  by  Percent  Complete 
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Figure  32  -  EAC  Slopes  by  Percent  Complete 
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Summary  of  Findings 

After  reviewing  the  results,  we  observed  definitions  #1  and  #2  did  not  provide  any 
useful  models.  These  results  indicate  that  definitions  #1  and  #2  are  not  significant  to 
predict  overall  contract  growth.  This  also  indicates  the  method  used  to  calculate  cost 
growth  by  definitions  #1  and  #2  is  not  effective  to  predict  overall  cost  growth. 

Definitions  #3  and  #4  had  much  better  results.  Definition  #3  had  fourteen 
significant  models.  In  addition,  it  revealed  that  level  three  EYM  data  was  predictive  of 
cost  growth  beginning  at  35%  with  an  adjusted  R2  of  .248569  and  at  45%,  the  R2 
increased  to  .988421.  Furthermore,  none  of  the  models  contained  both  level  three  and 
five  variables.  This  may  indicate  a  high  level  of  multicollinearity  between  the  two 
variables.  Stepwise  regression  selected  the  level  five  variable  once. 

Definition  #4  also  had  promising  results.  Twelve  of  the  thirteen  models  were 
significant  and  nine  of  them  contained  either  our  level  three  or  level  five  variable.  This 
model  revealed  that  level  five  EVM  data  became  predictive  of  overall  cost  growth  at  40% 
complete  with  an  adjusted  R2  of  .984166.  Similar  to  definition  #3  models,  none  of  the 
models  contained  both  level  three  and  level  five  variables. 

In  conclusion,  we  demonstrated  two  different  techniques  to  calculate  cost  growth 
at  lower  WBS  elements.  The  first  technique,  used  in  definitions  #1  and  #2,  involved  only 
the  change  in  individual  lower  level  elements.  This  technique  proved  ineffective  to 
predict  overall  cost  growth  and  did  not  provide  any  significant  results.  The  second 
technique,  used  in  definitions  #3  and  #4,  evaluated  the  cumulative  change  of  lower  level 
cost  data.  This  technique  did  provide  significant  results  as  indicated  earlier. 
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Our  results  also  reveal  level  three  and  level  five  EVM  data  becomes  predictive  at 
the  40%  complete  point  of  a  contract.  The  results  indicate  that  both  the  recalculated  EAC 
and  the  LRE  provided  by  the  contractor  are  predictive  at  approximately  the  same  point. 
However,  the  contractor’s  level  three  and  level  five  LRE’s  were  almost  identical.  Table 
25  and  Figure  3 1  identify  this  correlation. 

Our  results  also  indicate  level  three  EVM  data  is  a  better  predictor  of  contract 
cost  growth  than  level  five  EVM  data.  As  indicated  previously,  stepwise  regression 
chose  level  three  sixteen  times  and  level  five  only  three  times  in  the  forty  models 
between  definitions  #3  and  #4. 
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Chapter  5:  Conclusions 


Introduction 

In  this  chapter,  we  review  the  purpose  of  our  research  and  determine  if  our 
analysis  was  able  to  answer  our  research  questions.  We  also  discuss  some  limitations  and 
assumptions  we  had  throughout  our  research.  We  also  briefly  review  our  results  and 
findings  from  Chapter  Four  and  discuss  possible  implications  on  the  acquisition 
community.  We  conclude  this  chapter  suggesting  possible  follow-on  research  ideas. 

Review  Purpose  of  research 

The  purpose  of  our  research  was  to  determine  if  it  would  be  beneficial  for  the 
DoD  to  require  contractors  to  provide  level  five  EVM  data  instead  of  just  level  three.  In 
this  section,  we  evaluate  how  our  research  was  able  to  answer  our  research  purpose.  We 
also  look  at  our  research  and  investigation  questions  to  determine  if  our  research  was  able 
to  answer  them 

Our  first  research  question  was  if  level  five  data  is  more  predictive  than  level 
three  data.  To  answer  this,  we  decided  to  calculate  the  EAC  for  each  level  three  and  five 
element.  We  anticipated  from  previous  research  that  the  EAC  would  be  the  best  indicator 
for  contract  cost  growth.  We  also  determined,  though  limited,  sufficient  contracts  on  the 
DC  ARC  website  contained  level  five  EVM  data.  We  were  also  able  to  construct  a 
statistical  model  using  stepwise  and  multiple  regression  to  establish  a  relationship 
between  WBS  level  three  and  five  EVM  data  and  cost  growth  of  the  overall  contract. 

The  answer  to  our  first  research  question  is  WBS  level  five  EVM  data  is  not  more 
predictive  than  level  three. 
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Our  second  research  question  became  irrelevant  due  the  finding  of  our  first 
research  question.  Our  second  question  examined  the  benefits  of  level  five  EVM  data 
compared  to  the  cost  to  obtain  the  information.  However,  we  failed  to  discover  what  the 
additional  cost  would  be  to  obtain  this  data  because  we  discovered  level  five  EYM  data 
was  less  predictive. 

Limitations  in  our  Research 

Our  research  was  limited  in  two  key  ways.  The  first  limitation  dealt  with 
accessibility  to  EYM  data.  Due  to  the  requirements  of  our  research,  we  were  restricted  to 
the  EVM-CR  database.  The  main  reason  for  this  limitation  is  the  EVM-CR  is  the  most 
current  database  of  EVM  data  and  the  database  also  contained  contracts  with  lower  level 
EVM  data. 

Even  though  there  are  more  than  400  contracts  in  the  EVM-CR,  most  of  them  did 
not  meet  our  requirements.  Our  screening  criteria  included,  level  five  EVM  data  or 
lower,  contract  length  greater  than  twelve  months  and  complete  EVM  cost  data.  The 
database  contained  98  different  contracts  with  at  least  level  five  EVM  data.  However, 
many  of  those  contracts  were  either  missing  complete  EVM  data  or  were  not  greater  than 
twelve  months  in  length.  Our  screening  criteria  limited  us  to  only  forty  contracts. 

Assumptions  in  our  Research 

We  also  made  two  key  assumptions  in  our  research.  The  first  was  that  the  level 
five  EVM  data  was  not  biased.  This  is  a  concern  because  the  current  DoD  regulations 
require  contractors  to  provide  level  three  EVM  data  unless  a  contract  is  determined  high 
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risk  or  high  cost.  We  assumed  these  forty  contracts  represented  a  random  sample  of  the 
overall  contract  population,  not  just  high  risk  or  high  cost  contracts. 

The  second  assumption  necessitated  the  WBS  structures  contained  in  the  contracts 
were  CWBS,  not  PWBS.  We  made  this  assumption  because  WBS  level  three  of  a  PWBS 
is  equivalent  to  WBS  level  one  of  the  CWBS.  We  assumed  that  each  contract  contained 
the  CWBS  in  order  to  ensure  we  were  comparing  the  same  level  of  work  across  each  of 
the  contracts. 

There  needs  to  be  better  clarification  of  WBS.  For  example,  MIL  STD  881C, 
states  a  contractor  has  to  report  down  to  WBS  level  three.  Does  that  mean  a  contract  has 
to  report  down  to  level  three  for  each  contract  (CWBS)  or  just  that  the  contractor  is 
required  to  report  down  to  level  three  of  the  project  (PWBS).  The  difference  is  two 
additional  levels  of  data.  Level  three  of  the  CWBS  is  equivalent  to  level  five  of  the 
PWBS. 

Briefly  Review  Results 

We  were  able  to  establish  a  relationship  between  lower  level  WBS  EYM  data 
with  the  overall  contract  cost  growth.  However,  our  research  did  not  prove  WBS  level 
five  EYM  elements  are  more  predictive  than  WBS  level  three  EVM  elements.  Our 
research  proved  level  three  EVM  data  is  a  better  predictor  for  overall  contract  growth 
than  level  five  EVM  data.  Based  on  the  data  that  was  available  to  us,  it  would  not  be 
beneficial  for  the  DoD  to  require  contractors  to  provide  EVM  data  down  to  WBS  level 
five. 
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Follow-on  Research 


Based  on  our  research,  level  five  is  not  a  better  indicator  for  cost  growth  than 
level  three.  This  finding  contradicts  conventional  logic.  We  created  our  methodology 
and  statistical  models  without  any  bias  on  the  outcome.  However,  conducting  further 
research  using  a  larger  sample  size  to  either  confirm  or  disprove  our  findings  is  required. 

Our  research  posed  another  question  referencing  the  best  measurement  of  an 
EAC.  Should  a  contractor  provide  an  EAC  using  EVM  data?  Two  of  our  statistical 
models  used  the  contractors  EAC  and  two  contained  our  recalculated  EAC.  Both  the 
contractor’s  EAC  and  the  recalculated  EAC  proved  predictive  at  approximately  the  same 
percent  complete. 

However,  when  we  compared  the  level  three  EAC  value  with  the  level  five  EAC 
values,  the  contractor’s  EAC  showed  very  little  difference.  In  contrast,  the  recalculated 
EAC  for  level  three  and  five  were  completely  different.  A  follow-on  research  question 
entails  examining  the  requirements  of  a  contractor  to  provide  the  composite  EAC. 
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Appendix  A  -  Acronyms 

ACWP  (Actual  Cost  of  Work  Performed)  or  ACTUAL  COST  -  Cost  actually  incurred 


in  accomplishing  work  performed 

BAC  (Budget  At  Completion)  -  Total  budget  for  total  contract  thru  any  given  level 
BCWP  (Budgeted  Cost  for  Work  Performed)  or  EARNED  VALUE  -  Value  of 
completed  work  in  terms  of  the  work’s  assigned  budget 

BCWS  (Budgeted  Cost  for  Work  Scheduled)  or  PLANNED  VALUE  -  Time-phased 
Budget  Plan  for  work  currently  scheduled 

CA  (Control  Account)  -  Lowest  CWBS  element  assigned  to  a  single  focal  point  to  plan  & 
control 

EAC  (Estimate  At  Completion)  -  Estimate  of  total  Cost  for  total  contract  thru  any  given 
level  generated 

LRE  (Latest  Revised  Estimate)  -  Contractor’s  EAC 

MR  (Management  Reserve)  -  Budget  withheld  by  Contractor  PM  for  unknowns  /  risk 
management 

PMB  (Performance  Measurement  Baseline)  -  Contract  time-phased  budget  plan 
TAB  (Total  Allocated  Budget)  -  Sum  of  all  budgets  for  work  on  contract  =  NCC,  CBB, 
or  OTB 
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Appendix  B:  Contracts  Used  in  Analysis 


AAG  -  Advanced  Arresting  Gear  Program  (N68335-03-C-0205) 

Contractor:  General  Atomics  Corporation  Contract  Type:  CPFF 

WBS  Type:  Electronic  /  Automated  Software 
Service:  Navy  Phase:  RDT&E 

Beginning  and  Ending  Contract  Dates:  6/27/2008  -  5/27/2011 

AGM-88E  AARGM  -  AGM-88E  Advanced  Anti-Radiation  Guided  Missile  (AARGM) 
Program  (N00019-03-C-0353) 

Contractor:  Alliant  Techsystems,  Inc.  (ATK)  Contract  Type:  CPIF 

WBS  Type:  Missile 

Service:  Navy  Phase:  RDT&E 

Beginning  and  Ending  Contract  Dates:  3/31/2007  -  2/22/2009 

AMDR  -  Air  &  Missile  Defense  Radar  (N00024-10-C-5359) 

Contractor:  Northrop  Grumman  Corporation  Contract  Type:  FPIF 
WBS  Type:  Electronic  /  Automated  Software 
Service:  Navy  Phase:  RDT&E 

Beginning  and  Ending  Contract  Dates:  11/5/2010  -  9/29/2012 

B-2  MOP  -  Massive  Ordnance  Penetrator  (F33657-99-D-0028) 

Contractor:  Northrop  Grumman  Corporation  Contract  Type:  CPFF 

WBS  Type:  Aircraft 

Service:  Air  Force  Phase:  RDT&E 

Beginning  and  Ending  Contract  Dates:  9/25/2009  -  11/5/2010 

CEC  -  Cooperative  Engagement  Capability  (N00024-05-C-5100) 

Contractor:  General  Dynamics  Corporation  Contract  Type:  CPAF 

WBS  Type:  Electronic  /  Automated  Software 
Service:  Navy  Phase:  Prod 

Beginning  and  Ending  Contract  Dates:  3/30/2009  -  7/29/201 1 
Chem  Demil  -  CMA  (DACA87-89-C-0076) 

Contractor:  EG&G  Contract  Type:  CPAF 

WBS  Type:  Other 

Service:  Army  Phase:  PROD 

Beginning  and  Ending  Contract  Dates:  1/31/2000  -  5/26/2013 
Chem  Demil  -  CMA  (DAAA09-97-C-0025) 

Contractor:  Washington  Demil  Company  Contract  Type:  FFP 

WBS  Type:  Other 

Service:  Army  Phase:  PROD 

Beginning  and  Ending  Contract  Dates:  10/1/2004  -  5/24/2013 
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DDG  1000  -  ZUMWALT  CLASS  Destroyer  (N00024-05-C-5346) 

Contractor:  Raytheon  Company  Contract  Type:  CPAF 

WBS  Tyne:  Ship 

Service:  Navy  Phase:  RDT&E 

Beginning  and  Ending  Contract  Dates:  11/20/2005  -  3/31/2013 

DDG  51-  ARLEIGH  BURKE  CLASS  Guided  Missile  Destroyer  (N00024-02-C-2304) 
Contractor:  Northrop  Grumman  Corporation  Contract  Type:  FPIF 
WBS  Type:  Ship 
Service:  Navy  Phase:  Prod 

Beginning  and  Ending  Contract  Dates:  6/26/2005  -  5/30/2010 

EA-18G  -  Airborne  Electronic  Attack  variant  of  the  F/A-18  aircraft  (N00019-04-C- 
0005) 

Contractor:  The  Boeing  Company  Contract  Type:  CPAF 

WBS  Type:  Aircraft 

Service:  Navy  Phase:  RDT&E 

Beginning  and  Ending  Contract  Dates:  2/26/2004  -  2/26/2009 

EXCALIBUR  -  Family  of  Precision,  155mm  Projectiles  (DAAE30-98-C-1032) 
Contractor:  Raytheon  Company  Contract  Type:  CPIF 

WBS  Type:  Ordance 
Service:  Army  Phase:  PROD 

Beginning  and  Ending  Contract  Dates:  10/22/2006  -  2/24/2008 

EXCALIBUR  -  Family  of  Precision,  155mm  Projectiles  (W15QKN-08-C-0530) 
Contractor:  Raytheon  Company  Contract  Type:  CPIF 

WBS  Type:  Ordance 
Service:  Army  Phase:  RDT&E 

Beginning  and  Ending  Contract  Dates:  10/2/2011  -  8/26/2012 

FBCB2  -  Force  XXI  Battle  Command  Brigade  and  Below  Program  (W15P7T-04-D- 
G205) 

Contractor:  Northrop  Grumman  Corporation  Contract  Type:  CPAF 
WBS  Type:  Electronic  /  Automated  Software 
Service:  Army  Phase:  RDT&E 

Beginning  and  Ending  Contract  Dates:  12/31/2004  -  12/31/2009 

H-l  UPGRADES  (4BW/4BN)  -  United  States  Marine  Corps  Mid-life  Upgrade  to  AH- 

1W  Attack  Helicopter  and  UH-1N  Utility  Helicopter  (N00019-06-G-0001) 

Contractor:  Bell  Helicopter  Textron,  Inc.  Contract  Type:  CPFF 

WBS  Type:  Aircraft 

Service:  Navy  Phase:  RDT&E 

Beginning  and  Ending  Contract  Dates:  3/1/2008  -  5/8/2010 
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IAMD  -  Integrated  Air  &  Missile  Defense  (W31P4Q-08-C-0418) 

Contractor:  Northrop  Grumman  Corporation  Contract  Type:  CPIF 

WBS  Tyne:  Electronic  /  Automated  Software 

Service:  Army  Phase:  RDT&E 

Beginning  and  Ending  Contract  Dates:  9/3/2010  -  7/26/2013 

IS  PAN  -  Integrated  Strategic  Planning  and  Analysis  Network  -  Block  1  (FA8722-04-C- 
0009) 

Contractor:  Lockheed  Martin  Corporation  Contract  Type:  CPAF 

WBS  Type:  Electronic  /  Automated  Software 
Service:  Air  Force  Phase:  RDT&E 

Beginning  and  Ending  Contract  Dates:  10/30/2005  -  3/25/2011 

JAGM  -  Joint  Air-to-Ground  Missile  (W31P4Q-08-C-A123) 

Contractor:  Lockheed  Martin  Corporation  Contract  Type:  FPIF 

WBS  Type:  Missile 

Service:  Army  Phase:  RDT&E 

Beginning  and  Ending  Contract  Dates:  11/30/2008  -  5/30/2010 

JPALS  -  Joint  Precision  Approach  and  Landing  System  (N00019-08-C-0034) 
Contractor:  Rockwell  Collins,  Inc.  Contract  Type:  Other 

WBS  Type:  Electronic  /  Automated  Software 
Service:  Navy  Phase:  Other 

Beginning  and  Ending  Contract  Dates:  12/5/2008  -  7/2/2010 

JPALS  -  Joint  Precision  Approach  and  Landing  System  (N00019-08-C-0034) 
Contractor:  Raytheon  Company  Contract  Type:  CPIF 

WBS  Type:  Electronic  /  Automated  Software 
Service:  Navy  Phase:  RDT&E 

Beginning  and  Ending  Contract  Dates:  11/21/2008  -  12/31/2009 

JSOW  (BASELINE/UNITARY)  -  Joint  Stand-Off  Weapon  Baseline  Variant  and  Unitary 
Warhead  Variant  (N00019-05-G-0008) 

Contractor:  Raytheon  Company  Contract  Type:  CPFF 

WBS  Type:  Missile 

Service:  Navy  Phase:  RDT&E 

Beginning  and  Ending  Contract  Dates:  3/25/2007  -  4/1/2012 

JTRS  GMR  -  Joint  Tactical  Radio  System  Ground  Mobile  Radio  (DAAB07-02-C-C403) 
Contractor:  The  Boeing  Company  Contract  Type:  Other 

WBS  Type:  Electronic  /  Automated  Software 
Service:  Army  Phase:  Other 

Beginning  and  Ending  Contract  Dates:  9/29/2005  -  9/30/2010 
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LCS  -  Littoral  Combat  Ship  (N00024-03-C-2310) 

Contractor:  General  Dynamics  Corporation  Contract  Type:  CPIF 

WBS  Tyne:  Ship 

Service:  Navy  Phase:  RDT&E 

Beginning  and  Ending  Contract  Dates:  11/27/2005  -  4/4/2010 


LCS  -  Littoral  Combat  Ship  (N00024-11-C-2301) 

Contractor:  Austal  Contract  Type:  FPIF 

WBS  Type:  Ship 

Service:  Navy  Phase:  Prod 

Beginning  and  Ending  Contract  Dates:  7/27/2012  -  9/27/2013 

LPD  17  -  SAN  ANTONIO  CLASS  Amphibious  Transport  Dock  Ship  (N00024-04-C- 
2204) 

Contractor:  Northrop  Grumman  Corporation  Contract  Type:  CPIF 
WBS  Type:  Ship 

Service:  Navy  Phase:  PROD 

Beginning  and  Ending  Contract  Dates:  6/24/2007  -  12/20/2009 

MH-60R  -  Multi-Mission  Helicopter  Upgrade  (N00019-04-C-0130) 

Contractor:  Harris  Corporation  Contract  Type:  CPAF 

WBS  Type:  Aircraft 

Service:  Navy  Phase:  RDT&E 

Beginning  and  Ending  Contract  Dates:  1/26/2007  -  4/30/2010 

MH-60R  -  Multi-Mission  Helicopter  Upgrade  (N00019-08-C-0005) 

Contractor:  Telephonies  Corporation  Contract  Type:  CPIF 

WBS  Type:  Aircraft 

Service:  Navy  Phase:  RDT&E 

Beginning  and  Ending  Contract  Dates:  9/30/2008  -  4/30/2010 

MH-60R  -  Multi-Mission  Helicopter  Upgrade  (N00019-09-C-0059) 

Contractor:  L-3  Communications  Contract  Type:  FPIF 

WBS  Type:  Aircraft 

Service:  Navy  Phase:  Prod 

Beginning  and  Ending  Contract  Dates:  8/21/2009  -  11/26/2010 

MH-60S  -  Multi-Mission  Combat  Support  Helicopter  (N00019-03-G-0003) 

Contractor:  Sikorsky  Aircraft  Corporation  Contract  Type:  CPIF 

WBS  Type:  Aircraft 

Service:  Navy  Phase:  Unknown 

Beginning  and  Ending  Contract  Dates:  1/31/2006  -  3/31/2009 
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MPS  -  Mission  Planning  System  (FA8720-04-D-0005) 

Contractor:  The  Boeing  Company  Contract  Type:  CPAF 

WBS  Type:  Electronic  /  Automated  Software 
Service:  Air  Force  Phase:  RDT&E 

Beginning  and  Ending  Contract  Dates:  3/18/2011  -  10/5/2012 

MQ-4C  Triton  (Formerly  BAMS)  (N00019-08-C-0023) 

Contractor:  Northrop  Grumman  Corporation  Contract  Type:  CPAF 
WBS  Type:  UAV 

Service:  Navy  Phase:  RDT&E 

Beginning  and  Ending  Contract  Dates:  10/31/2008  -  5/27/2011 
P-8A  -  Poseidon  Program  (N00019-04-C-3146) 

Contractor:  Northrop  Grumman  Corporation  Contract  Type:  CPIF 

WBS  Type:  Aircraft 

Service:  Navy  Phase:  RDT&E 

Beginning  and  Ending  Contract  Dates:  6/26/2009  -  1/28/2011 
SM-6  -  Standard  Missile-6  (N00024-09-C-5305) 

Contractor:  Raytheon  Company  Contract  Type:  FPIF 

WBS  Type:  Missile 

Service:  Navy  Phase:  Prod 

Beginning  and  Ending  Contract  Dates:  7/25/2010  -  2/24/2013 
SM-6  -  Standard  Missile-6  (N00024-04-C-5344) 

Contractor:  Raytheon  Company  Contract  Type:  CPAF 

WBS  Type:  Missile 

Service:  Navy  Phase:  RDT&E 

Beginning  and  Ending  Contract  Dates:  12/31/2008  -  8/28/2011 

V-22  -  OSPREY  Joint  Advanced  Vertical  Lift  Aircraft  (N00421-10-D-0012) 
Contractor:  Raytheon  Company  Contract  Type:  CPAF 

WBS  Type:  Aircraft 
Service:  Navy  Phase:  Prod 

Beginning  and  Ending  Contract  Dates:  11/19/2010  -  4/26/2013 

V-22  -  OSPREY  Joint  Advanced  Vertical  Lift  Aircraft  (N61340-1  l-C-0004) 
Contractor:  Bell-Boeing  V-22  Program  Office  Contract  Type:  CPIF 
WBS  Type:  Aircraft 
Service:  Navy  Phase:  Prod 

Beginning  and  Ending  Contract  Dates:  5/31/201 1  -  10/31/2012 
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V-22  -  OSPREY  Joint  Advanced  Vertical  Lift  Aircraft  (N61339-08-D-0004) 
Contractor:  Bell-Boeing  V-22  Program  Office  Contract  Type:  CPFF 
WBS  Type:  Aircraft 
Service:  Navy  Phase:  Prod 

Beginning  and  Ending  Contract  Dates:  11/30/2009  -  10/31/2010 

VTUAV  -  Vertical  Takeoff  and  Land  Tactical  Unmanned  Air  Vehicle  (Fire  Scout) 
(N00019-00-C-0277) 

Contractor:  Northrop  Grumman  Corporation  Contract  Type:  CPIF 
WBS  Type:  UAV 

Service:  Navy  Phase:  RDT&E 

Beginning  and  Ending  Contract  Dates:  9/1/2006  -  12/31/2010 

VTUAV  -  Vertical  Takeoff  and  Land  Tactical  Unmanned  Air  Vehicle  (Fire  Scout) 
(N000 1 9-  12-C-0059) 

Contractor:  Northrop  Grumman  Corporation  Contract  Type:  CPIF 
WBS  Type:  UAV 

Service:  Navy  Phase:  RDT&E 

Beginning  and  Ending  Contract  Dates:  4/27/2012  -  6/28/2013 

WGS  -  Wideband  Global  SATCOM  Program  (FA8808-06-C-0001) 

Contractor:  The  Boeing  Company  Contract  Type:  FFP 

WBS  Type:  Space 

Service:  Air  Force  Phase:  PROD 

Beginning  and  Ending  Contract  Dates:  11/30/2006  -  12/20/2012 

WIN-T  Inc.  3  -  Warfighter  Information  Network  Tactical  Increment  3  (DAAB07-02-C- 
F404) 

Contractor:  General  Dynamics  Corporation  Contract  Type:  CPAF 

WBS  Type:  Electronic  /  Automated  Software 
Service:  Army  Phase:  RDT&E 

Beginning  and  Ending  Contract  Dates:  1/27/2012  -  12/28/2012 
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Appendix  C:  List  of  Independent  Variables  used  in  Model 


List  of  Variables  used  in  Statistical  Models 

1.  Total  Number  of  level  three  elements  in  contract 

2.  Total  Number  of  level  three  elements  with  EAC  data  in  contract 

3.  Total  Number  of  level  five  elements  in  contract 

4.  Total  Number  of  level  five  elements  with  EAC  data  in  contract 

Dummy  Variables  used  in  Statistical  Models 

1.  #  Lvl  5  elements  with  EAC  data  <  =  30 

2.  #  Lvl  5  elements  with  EAC  data  >  30  and  <150 

3.  #  Lvl  5  elements  with  EAC  data  >  =  150 

4.  RDT&E 

5.  Production 

6.  Other  Phase 

7.  Unknown  Phase 

8.  Army 

9.  Air  Force 

10.  Navy 

11.  Aircraft 

12.  Electronic/automated  software 

13.  Missile 

14.  Ordance 

15.  Ship 

16.  Space 

17.  UAV 

18.  Other  Handbook 

19.  CPAF 

20.  CPFF 

21.  CPIF 

22.  FFIP 

23.  Alliant  Techsystems,  Inc.  (ATK) 

24.  Austal 

25.  Bell-Boeing  V-22  Program  Office 

26.  Bell  Helicopter  Textron,  Inc. 

27.  EG&G 

28.  General  Atomics  Corporation 

29.  General  Dynamics  Corporation 

30.  Harris  Corporation 
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31.  L-3  Communications 

32.  Lockheed  Martin  Corporation 

33.  Northrop  Grumman  Corporation 

34.  Raytheon  Company 

35.  Rockwell  Collins,  Inc. 

36.  Sikorsky  Aircraft  Corporation 

37.  Telephonies  Corporation 

38.  The  Boeing  Company 

39.  Washington  Demil  Company 

40.  Top  3  DoD  Contractor 

41.  Top  5  DoD  Contractor 

42.  Contract  length  12-24  Months 

43.  Contract  length  24-36  Months 

44.  Contract  length  more  than  36  months 

45.  Percent  of  program  contract  covered  <25% 

46.  Percent  of  program  contract  covered  25%  -  50% 

47.  Percent  of  program  contract  covered  50%  -  75% 

48.  Percent  of  program  contract  covered  more  than  75% 
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Appendix  D  -  .IMP®  Output  Screens  for  Growth  Definition  #1 

Growth  Definition  #1  Bin  -  5% 


■C  Thesis  JMP  File  -  Response  Variable  #1  -  Overlay  Plot  of... 


Rows 


90 


Growth  Definition  #1  Bin  - 10% 


>C  Thesis  JMP  File  -  Response  Variable  #1  -  Overlay  Plot  of- 


Rows 


fcW  □  ▼ 


91 


Growth  Definition  #1  Bin  - 15% 


C  Thesis  JMP  File  -  Explanatory  Variable  #1  -  Overlay  Plot .... 


92 


Growth  Definition  #1  Bin  -  20% 


C  Thesis  JMP  File  -  Explanatory  Variable  #1  -  Overlay  Plot . 


_  n 


^©Overlay  Plot 


< 


0.3 

0,25 

0.2 

0.15 

0.1 

0,05 

0^ 

-0,05 


N#  * 


—r~ 

10 


—r~ 

15 


— r~ 

20 


— r~ 

25 


l — 

30 


I — 

35 


— I — 

40 


45 


Rows 


93 


Growth  Definition  #1  Bin  -25% 


94 


Growth  Definition  #1  Bin  -  30% 


95 


Growth  Definition  #1  Bin -35% 


XT-  Thesis  JMP  File  -  Explanatory  Variable  #1  -  Overlay  Plot  _ 


4  0. 


^Overlay  Plot 


£  ^ 


0.0S 


£  0.06 


'r!  U 
£  < 


0,04 


0.02 


-0.02 


Rows 


96 


Growth  Definition  #1  Bin  -  40% 


97 


Growth  Definition  #1  Bin  -45% 


98 


Growth  Definition  #1  Bin  -  50% 


99 


Growth  Definition  #1  Bin -55% 


K'-  Thesis  JMP  File  -  Explanatory  Variable  #1  -  Overlay  Plot ... 


□  ▼ 

100 


Growth  Definition  #1  Bin  -  60% 


101 


Growth  Definition  #1  Bin -65% 


102 


Growth  Definition  #1  Bin  -  70% 


■C  Thesis  JMP  File  -  Explanatory  Variable  #1  -  Overlay  Plot ... 


n 


x 


A 

(^Overlay  Plot 

0.2 


0.15 


0.1 


0.05 


~T~ 

15 


T 

20 


~~r 

25 


Rows 


~T~ 

B0 


~T~ 

B5 


~r 

40 


45 


^3  ; 

103 


Growth  Definition  #1  Bin  -75% 


>C  Thesis  JMP  File  -  Explanatory  Variable  #1!  -  Overlay  Plot ... 


❖  E»  □  ▼ 

104 


Growth  Definition  #1  Bin  -  80% 


V-' 


C  Thesis  JMP  Fife  -  Explanatory  Variable  #1  -  Overlay  Plot . 


Overlay  Plot 


m  0.15 


£  - 
<u 


0.1 


0.05 


10  15  20  25  30  35  40  45 

Rows 


105 


Growth  Definition  #1  Bin -85% 


106 


Growth  Definition  #1  Bin  -  90% 


X^  Thesis  JMF  File  -  Explanatory  Variable  #1  -  Overlay  Plot ... 


_  n 


-^©Overlay  Plot 


> 

0.2- 

'Ll 

cn 

c 

ra 

JZ 

0.15 

£ 

'Ll  1— 

u  u 

0.1 

4— 

c 

* 

Q 

0.05 

* 

o 

o 

0 

# 

*  t  *  * 


~r 

10 


15 


~r 

20 


25 


Rows 


~r 

30 


35 


~r 

40 


45 


□  ▼ 

107 


Growth  Definition  #1  Bin  -95% 


108 


Growth  Definition  #1  Bin  - 100% 


109 


Appendix  E  -  ,TMP®  Output  Screens  for  Growth  Definition  #2 
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Appendix  F  -  .IMP®  Output  Screens  for  Growth  Definition  #3 
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Appendix  G  -  .IMP®  Output  Screens  for  Growth  Definition  #4 
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-0.24 

0.8249 

-0.05855 

4.3996159 

CPAF 

-0.20026 

0.416466 

-0.48 

0.6557 

-0.09521 

3.1237354 

CPIF 

0.1033562 

0.394512 

0.26 

0.8063 

0.055228 

3.5407277 

Austal 

-0.037358 

0.948691 

-0.04 

0.9705 

-0.00815 

3.4124917 

Bell-Boeing  V-22  Program  Office 

0.2358035 

0.823-681 

0.29 

0.7889 

0.051439 

2.5724049 

EGSiG 

0.037667 

0.930557 

0.04 

0.9697 

0.008217 

3.2332771 

Harris  Corporation 

1.3823297 

0.827161 

1.67 

0.1700 

0.301547 

2.5941895 

L-3  Communications 

-0.214506 

0.88916 

-0.24 

0.8212 

-0.04679 

2.9976513 

Rockwell  Collins,.  Inc. 

-0.412871 

0.912154 

-0.45 

0.6743 

-0.09007 

3.1546988 

Telephonies  Corporation 

-0.333631 

0.725644 

-0.46 

0.6696 

-0.07273 

1 .9964961 

Top  5  DOD  Contractor 

0.1458827 

0.683076 

0.21 

0.8413 

0.077951 

10.614773 

24-36 

-0.945304 

0.4598 

-2.06 

0.1090 

-0.23549 

1 .5364069 

Lvl  5  Cum  Change/ Final  Lvl  1  EAC 

35%  5.4204906 

1.180813 

4.59 

0.0101* 

0.373975 

2.8881353 

Effect  Tests 


Effect  Details 
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Growth  Definition  #4  Bin -40% 
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•C  Thesis  JMP  File  -  Explanatory  Variable  #4  -  Overlay  Plot . 
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Growth  Definition  #4  Bin -45% 
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'■C  Thesis  JMP  File  -  Explanatory  Variable  #4  -  Overlay  Plot 


Growth  Definition  #4  Bin  -  50% 
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Growth  Definition  #4  Bin -55% 
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Growth  Definition  #4  Bin -60% 
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C  Thesis  JMP  File  -  Explanatory  Variable  #4  -  Overlay  Plot ... 
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Growth  Definition  #4  Bin -65% 
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Growth  Definition  #4  Bin  -  70% 
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Growth  Definition  #4  Bin  -75% 
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X;  Thesis  JMP  File  -  Explanatory  Variable  #4  -  Overlay  Plot ... 
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Growth  Definition  #4  Bin -80% 
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-  Thesis  JMP  File  -  Explanatory  Variable  #4  -  Overlay  Plot . 


15  20  25 

Rows 


□  ▼ 

Growth  Definition  #4  Bin -85% 
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Growth  Definition  #4  Bin  -  90% 


‘C  Thesis  JMP  File  -  Growth  Definition  #4  -  Overlay  Plot  of ... 
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Growth  Definition  #4  Bin  -95% 


>C  Thesis  JMP  File  -  Explanatory  Variable  #4  -  Overlay  Plot . 
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Growth  Definition  #4  Bin  - 100% 
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Appendix  H  -  Relationship  between  Level  Three  and  Level  Five  Data  (Using 


Contractor’s  EAC  or  LRE 


Comparing  Level  3  with  Level  5  at  Bin  -  5% 
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Comparing  Level  3  with  Level  5  at  Bin  - 10% 
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Comparing  Level  3  with  Level  5  at  Bin  - 15% 
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Comparing  Level  3  with  Level  5  at  Bin  -  20% 
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Comparing  Level  3  with  Level  5  at  Bin  -  25% 
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Comparing  Level  3  with  Level  5  at  Bin  -  30% 


180 


Comparing  Level  3  with  Level  5  at  Bin  -  35% 
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Comparing  Level  3  with  Level  5  at  Bin  -  40% 
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Comparing  Level  3  with  Level  5  at  Bin  -  45% 
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Comparing  Level  3  with  Level  5  at  Bin  -  50% 
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Comparing  Level  3  with  Level  5  at  Bin  -  55% 
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Comparing  Level  3  with  Level  5  at  Bin  -  60% 
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Comparing  Level  3  with  Level  5  at  Bin  -  65% 
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Comparing  Level  3  with  Level  5  at  Bin  -  70% 
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Comparing  Level  3  with  Level  5  at  Bin  -  75% 
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Comparing  Level  3  with  Level  5  at  Bin  -  80% 
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Comparing  Level  3  with  Level  5  at  Bin  -  85% 
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Comparing  Level  3  with  Level  5  at  Bin  -  90% 
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Comparing  Level  3  with  Level  5  at  Bin  -  95% 
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Comparing  Level  3  with  Level  5  at  Bin  - 100% 
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Appendix  I  -  Relationship  between  Level  Three  and  Level  Five  Data  (Using 

Recalculated  EAC) 

Comparing  Level  3  with  Level  5  at  Bin  -  5  % 
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Comparing  Level  3  with  Level  5  at  Bin  - 10% 
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Comparing  Level  3  with  Level  5  at  Bin  - 15% 
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Comparing  Level  3  with  Level  5  at  Bin  -  20% 
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Comparing  Level  3  with  Level  5  at  Bin  -  25% 
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Comparing  Level  3  with  Level  5  at  Bin  -  30% 
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Comparing  Level  3  with  Level  5  at  Bin  -  35% 
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Comparing  Level  3  with  Level  5  at  Bin  -  40% 
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Comparing  Level  3  with  Level  5  at  Bin  -  45% 
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Comparing  Level  3  with  Level  5  at  Bin  -  50% 
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Comparing  Level  3  with  Level  5  at  Bin  -  55% 
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Comparing  Level  3  with  Level  5  at  Bin  -  60% 
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Comparing  Level  3  with  Level  5  at  Bin  -  65% 
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Comparing  Level  3  with  Level  5  at  Bin  -  70% 
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Comparing  Level  3  with  Level  5  at  Bin  -  75% 
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Comparing  Level  3  with  Level  5  at  Bin  -  80% 
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Comparing  Level  3  with  Level  5  at  Bin  -  85% 
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Comparing  Level  3  with  Level  5  at  Bin  -  90% 
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Comparing  Level  3  with  Level  5  at  Bin  -  95% 
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Comparing  Level  3  with  Level  5  at  Bin  - 100% 
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